A Beta That Started With Three Testers, and the Rule That Decided

Tomorrow night the Duskglow beta starts with three people. The plan I wrote in early October asked for eight to twelve, with six finishing the two weeks as the floor. I'm not going to explain the gap away. In the same plan, before a single invite went out, I wrote down what would happen if I fell short, and this week those rules got tested twice, once by my own neglect and once by me trying to rewrite one of them the day before it was due.

Two rules, written before the data

Everything in the beta hung on one question. Does an adult who has trouble winding down at night get through setup on their own phone and come back over two weeks, without me there to help? That is a retention question, and a retention number only means something if everyone is measured the same way. So the plan was one group, one shared fourteen-night window, and a build that stayed frozen the whole time.

I had looked at the looser version, letting people in as they turned up and shipping as I went, and turned it down for two reasons. It would mix results from different builds. And it would mostly tell me what felt off, which the first round of testing had already told me.

Then I wrote two rules, because I know how I behave when a number disappoints me. If fewer than six people had confirmed by Thursday, October 29, one post would go out in a community whose rules allowed it. If fewer than six had confirmed by Sunday, November 1, I would run the observed first sessions only and put the retention read off. The phrase in the plan is "deferred, not run thin."

The rule nobody ran

Thursday came and went.

Two people had confirmed, and nobody ran the check, me included. My attention was on getting the code ready for a friend's review, and the date slid past. The post went out two days late.

A rule written down is a decision made in a calmer moment. It only binds you if something puts it in front of you on the day. I had put the date in a document, and documents don't interrupt anyone.

Fourteen applications in one night

Within hours the post brought in fifteen applications. One was a duplicate. Nine of them arrived inside twelve minutes at two in the morning Pacific, on a post with two upvotes and no comments.

My first feeling was relief, and I noticed it as it was happening, because relief is exactly the feeling that makes you stop checking. Fourteen people would have taken me past the floor in one night. So before inviting anyone I sent all fourteen a short follow-up with two questions, worded as part of the application: how did the link reach you, and what do your last ten minutes before sleep look like? All fourteen answered within fourteen minutes of each other. All of them said Reddit. Every reply also carried one detail that contradicted what the application said.

The most likely reading was a group working the post for the small thank-you gift card. I declined all fourteen and took the post down.

I'd defend this call to anyone who asks why I turned away a full cohort. A retention read on people who joined for a gift card measures the gift card. It would have produced a number, the number would have looked like an answer, and I would have planned the next quarter around it. An empty seat costs time. A wrong seat costs the conclusion.

Rewriting the rule the day before

Now the part I'm less proud of. The day before the second rule was due, I rewrote it. The new version let each later tester start on their own night, any night through November 8, with their own fourteen nights. It kept the build frozen until the last of those windows closed, and it said a result under six would be reported as directional.

On paper the reasoning held up. A frozen build removes the mixed-builds problem, and the curve I measure was already per person. But I can see the shape of what I was doing. I had a rule that said stop and defer, I didn't want to stop, and so the rule got longer. Once the forum applicants fell away, the amended version had nothing left to stand on, and on Sunday the original rule fired.

So three people start tomorrow as observed testers, the two who had confirmed and a third, someone from the first round, who said yes that same weekend. Where people agreed to it, I'll watch their first session over a screen share. Then I read every reply to the night-one and night-three notes and write it up on November 20. The retention read waits for a second window that opens only once six people say yes. Realistically that's January.

What the freeze was for

The freeze went on Friday, with the last of the cleanup before my friend's review. My friend had looked at the repository and found more lines than expected, so I measured: about 58,000, roughly a third of them tests. Cutting more files wouldn't have moved that much, and a reviewer's time is the scarce part. The last change was a guide instead, six boundaries in the order I'd read them, one question for each, a table of where each one is proven, and a list of what to skip.

It came off two days later. Its reason had been the retention read, one build for everyone so the numbers compare. With the read deferred, the only argument left was safety, and the merge gate already covers most of that with a type check, a build, 1,175 tests, and a phone check whenever the screen changes. A blanket "fixes only" added almost nothing the gate didn't. The one gap I could find, I kept: a change to the night flow on a night a tester is likely to use it gets the phone check first.

What the decision missed was small and telling. The instructions my coding agent reads at the start of every session still said the build was frozen. A premise lives in every place that repeats it, the plan being only the first, and an agent would follow a stale instruction as faithfully as a current one. Fixing that file was the next session's first job.

Mental Models

A rule only binds if something reads it on the day. Writing the threshold down was the easy half. My first rule lapsed because its date lived in a document and nothing put it in front of me that Thursday.

Turn away the wrong cohort, even when it is the only one on offer. Fourteen applicants would have cleared the floor, most likely to produce a confident number about a gift card. Declining them kept the question intact.

Watch for a stop rule getting longer. When a rule is about to fire and you find yourself amending it, ask whether the amendment answers a new fact or just moves the stop. Mine lasted a day.

Retire a rule when its reason goes, then go find every copy of it. The freeze existed for comparable data. When the comparison was deferred the freeze went too, and the instruction that repeated it survived one more session.