Shipping the change I could not take back

Today I merged the encryption work into production. I have been more careful about this change than anything else I have shipped on Duskglow, for two reasons that happen to point the same way. It fixes a real problem for the people using the app. And it is close to the hardest thing in the project to undo once it is out.

The problem first. For months, someone could be partway through a conversation, their phone could lock or the browser could quietly drop the tab, and the words they had written could be gone. Not a great look for a journaling app. The work I shipped moves in-progress writing into client-side encryption, and in the same motion it closes the gap where that writing used to disappear. One change, two jobs. A privacy improvement and a data-loss fix arriving together.

Now the careful half. This did not go out to a handful of new accounts. It reached everyone at once, and an encryption rollout is not the kind of thing you quietly revert the next morning when you find a problem. That fact shaped how I shipped it more than any single line of code did.

Verification should scale with how hard it is to undo

Most of what I ship is cheap to take back. A copy tweak, a layout fix, a setting that defaults the other way. If it is wrong I revert it and almost nobody notices. This was not that.

So I made the checking heavier than the change looked like it needed, on purpose. Four separate gates had to pass before I was allowed near the production branch. An automated test gate that has to go green on every browser engine I support. The full adversarial safety suite, run against the live model, the same prompt-injection and crisis-handling probes I run before anything that touches that path. A clean build and type check. And then the slow one no script covers, where I signed in and used the feature the way a person actually would.

I will be honest that the first three are easy to treat as box-ticking. The real discipline is deciding the weight in advance. A change you can undo in thirty seconds earns a light pass. A change you cannot really undo earns all of this, and you make that call while you can still think straight, not in the middle of whatever goes wrong.

The friction in the approval step is the point

I run the production merge behind a rule. It needs my explicit approval at the moment it happens, every time, and there is no "stop asking me" version of that approval anywhere in the setup.

Turning it off would be faster. I keep it on for one reason. That pause between "everything looks green" and "push it" is exactly where I catch the thing I had talked myself past an hour before. On the step you cannot reverse, speed is not what I am optimizing for. I want one last real look from someone who has to actively say yes.

Friction like that gets called bureaucracy, and on reversible steps the complaint is fair. I have stripped plenty of it out elsewhere. On the one-way steps I have come to see the friction as something I am paying for deliberately, and the morning I route around it to save two minutes is the morning it stops protecting me.

"Deployed" and "working" are not the same claim

A couple of minutes after I pushed, the deploy dashboard went green. That tells me the build shipped. It does not tell me the feature works for a real human, and I have been caught by that gap before.

So I did not stop at the checkmark. I opened the live app as an ordinary user and went looking for actual signal. Telemetry confirmed the new version was the one running. A real session set up its encryption cleanly and logged a clean open-to-close lifecycle with nothing erroring along the way. The part I most wanted to see was the key setup working in the fresh production environment, off my own machine where it had always behaved, and it did.

That last one carried the most weight, because "works on my laptop" is the oldest lie in the job. Watching it work somewhere that is not my laptop is the evidence I trust.

Decide the failure response while you are calm

I went in with a plan for what I would do if it looked wrong after going live, and the plan was not to erase what I had shipped. On a history that other work is already built on top of, rewriting the past does more damage than the problem it is meant to fix. If I needed to back the change out, I would do it by shipping a forward change that reversed the behavior, leaving the record intact.

I never had to. The thing I keep taking from days like this is that I made the decision while I was calm, before anything was on fire. The version of me in the middle of an incident is not the one I want choosing between the clean fix and the destructive shortcut.

Mental models

Verification scales with reversibility. A change you can undo in seconds earns a light check. One you cannot really undo earns a heavy one. Decide which you are shipping before you start, not after it breaks.

Friction on irreversible steps is a feature. Per-action approval with no permanent bypass is slower by design. The pause is where you catch what you rationalized away. Save the shortcut for the changes you can actually take back.

"Deployed" is not "working." A green deploy means the build went out. Whether it works for a real person is a separate question, and you answer it with live signal, not with the absence of red.

Pre-commit the calm decision. Choose your failure response before you deploy, while you can still reason clearly. The person handling an incident as it unfolds should not be the one inventing the recovery plan.