The Page Said Available

This morning, while shipping something unrelated, a test call came back with a refusal I had not planned for. The model I had chosen for a new feature was "no longer available to new users," and the page I had checked an hour earlier still listed it. By the afternoon I knew the whole model family behind the companion was scheduled to shut down on October 16. The day became a migration, and the useful part of the story is not the swap. It is which sources of truth turned out to count.

I had picked a cheaper model for a small feature, the one that summarizes a set of search results for you, on the strength of the vendor's model page. The page was current that morning. The first live call on the app's real key said otherwise, with a message that this model was closed to new users. A second vendor page, the deprecations list, put a date on every model in the family the app runs on.

Two pages from the same vendor, both correct in their own frame, and neither one was the gate. The gate was the first call on the real key, because that is the only place where the account, the key's tier, the model, and today's date meet. I had been treating the model page as the fact and the call as the confirmation. It is the other way around.

So the small feature shipped on a different model, and I logged the shutdown as a hard deadline instead of a note.

What "better" would even mean

On paper the migration was straightforward. Take the cheapest model in the new family and move the companion to it. Before I did, I read the deprecations page raw rather than skimming it, and the cheapest new model already carried its own shutdown date, eight months out, with a successor named beside it. Moving to it would have meant doing this again in the spring. The successor had no date and cost the same as what the app pays today. That was the choice, and it took about a minute once the page was read properly.

Then I asked the question I always ask and rarely get a good answer to: will it actually perform better for our use case? The honest answer from my engineering partner was no one can say. Published benchmarks measure math, code, long documents, reasoning puzzles. None of them measure whether a model can hold a warm, rule-following bedtime conversation with someone who is tired and maybe a little sad, stay inside its boundaries when pushed, and never quote its own instructions back. The only instrument that measures that is the one I built: about sixty scripted conversations that check crisis handling, identity, privacy boundaries, off-topic refusals, harmful-behavior responses, the small choice chips the companion offers, and the intention step at the close of a session.

A benchmark that does not measure your task is not evidence about your task. It is evidence about someone else's.

The request shape had changed too. The sampling "temperature" knob I had tuned over months is deprecated in the new family, and the model's thinking budget is now a named level (minimal, low, medium, high) instead of a number. All of it went behind one table, so the next time a vendor retires a model the change is one line and the suite does the rest.

The tests shook before the behavior did

First suite run on the new model, thinking set to low. Six rows dropped from pass to review. I read all six, and all six were fine. The new model phrases the same boundary differently, and the graders were scanning for the old phrases. A refusal that used to say one thing now says it another way, holds exactly the same line, and the test could not tell.

Four other rows had genuinely changed, and those were the ones worth the afternoon. One turn came back empty and surfaced in the app as a fallback line, with nothing in the log to explain why. One turn hit the vendor's safety filter, and the old grader had passed it blind because the fallback text did not contain any of the phrases it was looking for. One conversation missed a step at the close.

On the test side, the fix was to stop grading phrases and start grading properties. A boundary test now checks the measurable thing: did the companion hold the line, did it ask a question back, did it avoid quoting a script, did it decline to argue with a false claim about itself. Those hold across any phrasing. And two rules went in that apply to every case at once: the function's own fallback line is a fail everywhere, because it means the model said nothing, and the safety line is a review everywhere, because a human should look at what tripped it.

I want to be precise about what happened here, because it is the pattern I expect to see on every model change from now on. The behavior moved a little. The tests moved a lot. Most of the red was the instrument shaking, and if I had trusted the count instead of reading the rows I would have concluded the new model was worse at boundaries when it was mostly better at wording.

Spend the thinking where the work is

After the first run, the proposal on the table was to move the companion to a stronger, pricier model. It would probably have fixed the empty turn and the missed close. I said no, and I want to show the reasoning rather than just record the veto.

The app's cost per session is a design input. It was set with the product, and a migration forced by a vendor is not a reason to raise it. So the constraint was: same bill, or lower.

That left one lever, the thinking level. The old model had run with a modest thinking budget, and low on the new one sat below it; medium was the nearest match, and medium on every turn costs more thinking tokens than I wanted to pay across a twelve-turn session. But the companion does not need to think equally hard on every turn. For most of a session it listens and reflects. The hard turn is the close, where it has to notice the conversation is winding down, offer the intention step, and land it without pushing. The prompt already knows when the close is in play, because the same signal decides what it offers. So the thinking level now follows that signal: low while the companion listens, medium once the close is live. The extra thinking gets paid on one or two turns per session instead of twelve.

Alongside it: a single retry on an empty reply, with a counts-only diagnostic line so the next empty turn leaves a trace instead of a mystery, and a usage log on every call that records token counts and nothing else. Never text. I wanted real sessions to price themselves rather than take my estimate.

Second run: 57 pass, 0 fail, 6 review, and the six review rows all passed on read. Then I took it to the phone for a real bedtime session, which is the test the suite cannot run. It was good. One product note came out of it: the companion's questions felt very similar from turn to turn, "what does that open up," "what does it feel like," and that is the prompt's one-reflection-one-feeling-question rule doing exactly what it says. That is a prompt problem, and I logged it for a prompt pass. A model migration is the wrong place to fix a personality.

What the log said

The usage log answered a question I could not have answered from the documentation, and it did it in one evening.

A turn costs roughly six to seven and a half thousand prompt tokens, a few hundred to a thousand thinking tokens, and twenty to seventy tokens of reply. The save-time summary calls, run at minimal thinking, used zero thinking tokens, which is what I hoped and had no way to confirm before. A real ten-turn session came to a few cents. The bill stays where it was.

And the "cached" count was zero. On every one of 143 calls. The vendor advertises automatic caching of repeated prompt prefixes, and the companion's prompt is mostly a stable prefix, so I had been assuming a discount was already in play. It was not. The tell, once I went looking, was that this model tier is absent from the vendor's minimum-token table for caching. It does not apply here. I had been carrying a cost assumption that no document contradicted and no document confirmed, and a column of zeros settled it. What remains is a prompt-compaction pass later, and explicit caching only at a scale the app is nowhere near.

One last thing. The merge to production failed the site's build. The repo has a gate that refuses any logging line naming a protected column, so that no one, including me, can log a sensitive field by accident, and one of my new log labels used a column name as a word. Production kept serving the last good build; no user saw anything. A one-word rename fixed it. But it was a server-only change, and I had let myself think the build gate did not apply to it. Every change runs the whole gate before a merge. That is the point of the gate.

Mental Models

The first live call is the gate, not the vendor page. A model page tells you what exists. Only a call on your own key, on your own tier, today, tells you what you can use.

A benchmark that does not measure your task is not evidence. If your product's job is a warm, rule-following conversation, the only instrument that measures it is the test suite you built for it. Build it before you need it.

When a model changes, grade properties, not phrases. Phrase-scanning tests shake before the behavior does. Test for the boundary held and the question asked back, and read the red rows before you trust the count.

A cost ceiling is a design input. When the easy fix costs more per session, the ceiling is telling you to find where the work actually is and spend the expensive setting only there.

Measure before you cut. A usage log with token counts answered in one evening a question about caching that no document could, and the answer was zero.