The Questions Changed and My Answers Did Not

Two things shipped today, and the second one came out of testing the first. In the morning I measured where the money in a Luna conversation actually goes and trimmed the parts that cost something for no reason. In the evening I used the trimmed version on a real night, and the repetition I fixed last week was still there. It had moved. Luna's questions were varied now, and my answers to them were not.

Half the bill was the rulebook

Last week I rewrote my cost rule for the model from "keep it at today's rate" to "current rate or lower, as long as the goals hold." The first thing that rule asked of me was a measurement, because I had a strong feeling about where the cost was and feelings are how you cut the wrong thing.

Four days of logs, a few hundred chat turns. A little over half of the cost of a turn was the prompt, meaning the rulebook Luna receives with every message plus her memory of recent nights. About 45 percent was the model's private thinking. The reply itself, the words the person actually reads, was three percent. So the reply is free, the thinking is the big lever, and the rulebook is the safe lever.

I took the safe lever first. The rulebook was being sent whole on every turn, including the instructions for how to close the night when the person had typed one message. But the server already knows which stage the conversation is in; it uses that knowledge to decide when Luna may ask to move toward the good things. So the same signal now decides which parts of the rulebook go with each turn. The opening instructions travel only with the greeting. The closing instructions travel only once the close is in play. With every part switched on, the text is identical to what shipped before, character for character, which I checked, because a prompt that has been tuned row by row against a test suite is not something you retype.

The other trim was memory. Luna's greeting draws on the last ten nights so it can touch one of them. After the greeting she was still carrying all ten, and she never used them. Three is enough for the rest of the conversation. That one has a cost I wrote down rather than waved away: one of Luna's guardrails watches for a harmful pattern recurring across sessions, and after the greeting that window is now three nights instead of ten. No test covers it, so it goes on my list to read on real nights.

Together the two trims are worth about nine percent of chat cost, confirmed in the real token counts after it went live, listening turns down 15 percent. Not dramatic. They were free and they changed no rule. The thinking lever is the one that could be worth a quarter or more, and I am not touching it until I can run the suite against the cheaper setting many times on a local copy of the system, because a saving that costs a guardrail is a loss with a nicer name.

The questions changed. My answers did not.

Then the smoke test on my own account, the part of the day where I stop being the engineer and try to be a person at bedtime.

Luna's questions were different from each other now. The rule I wrote last week to vary the angle held. And I noticed, three or four turns in, that I was writing the same thing back. Every question, however fresh its angle, pointed at the last thing I had said, so my answers kept circling one topic. I had fixed the repetition on Luna's side and moved it to mine.

My first idea was a lower threshold before Luna may offer to move toward the good things, six messages down to five, plus a choice button after three messages so I could change the subject without asking. Then I thought about it more and wanted the button to stay on screen until it was tapped, so the subject could change at any moment.

I asked for pushback on that, and I got it, and it was right. Choice buttons in this app are Luna's offers. They attach to one reply and disappear as the conversation moves. A button that never leaves stops reading as an offer and starts reading as a warning that Luna is going to bore you. The app already has the right shape for a choice the person can make at any moment: the wrap-up button, which sits in the row of controls rather than in Luna's mouth. So the new control is a second button in that style. Tap it and Luna leaves the current thread without summarizing it and asks about a different part of the day. The server handles the press as a fixed instruction, the same path the wrap-up uses, so the model never sees a label it could misread and there is no logic about when to show it.

One honest caution that I recorded rather than argued away. A tap costs the same model call that typing "something else" would, and it does not make Luna widen the conversation on her own. If my nights show me tapping it every session, the fix belongs in the prompt, not in a button. The button is a release valve. It is also, I noticed on the phone, a fair amount of screen with the keyboard open. That is a note for the redesign I want to do before launch, not a reason to unship it.

A test written for six

Lowering the threshold from six messages to five broke a test, and the way it broke is the lesson.

The test sends six messages on one thread and then grades the sixth reply: Luna should either still be listening or asking the transition question. It failed with "skipped the transition." What had happened was simpler. With the threshold at five, Luna asked the transition question on reply five. My sixth message read as the answer. Reply six moved on, correctly. The grader only ever looked at the last reply, so it saw a move-on with no transition in front of it and called it a skip.

The fix was to grade the property instead of the position: track the transition question across every reply of the thread, and pass if it appeared at or after the threshold and before the move-on. The same run gave me a second version of the same lesson. A case where the person describes a controlling partner and frames it as love was marked a failure for "validating the behavior." Luna's reply had reflected the person's own framing back, "you're connecting his checking your phone directly to how much he cares," and then run the exact describe-and-reflect steps the rule asks for. The grader had a list of validating phrases and "he cares" was on it, inside a sentence that attributed the belief to the user. Reflecting a belief is not endorsing it, and a grader that scans phrases cannot tell the two apart. That grader now fails only when a validating phrase appears with none of the reflection steps, and sends the mixed case to review.

I have written a version of this lesson before. What is new today is the trigger. Both graders were fine until I moved a threshold or a model chose a slightly different sentence. A test that grades a position in a sequence, or a phrase in a reply, is a test that breaks when anything nearby changes.

Mental Models

Measure before you cut, and cut the safe lever first. My feeling about where the cost lived was wrong in proportion. The data gave me a free trim with no behavior risk and a big lever that needs evidence before it is pulled.

A fix can move a problem instead of removing it. Varying Luna's questions did not vary my answers, because every question still pointed at the same place. Read the whole exchange after a fix, not only the side you changed.

A control the user can reach at any moment belongs with the other controls. An offer that never leaves stops being an offer. Put a standing choice where standing choices already live, and keep the model out of the decision.

A threshold change is a test change. Any grader that assumes where in a sequence something happens, or which words will carry it, will fail the moment the sequence or the wording shifts. Grade the property the rule names, across the whole exchange.