The Test That Only Fails in Production
You've written the test. It passes locally. It passes in CI. It passes in staging. You deploy. Within four hours, a user hits a state your test never considered, and the thing breaks in a way that's embarrassing to explain in the post-mortem.
This isn't a story about bad tests. It's a story about the gap between the environment you model and the environment that exists.
The Model Is Not the Territory
Every test suite is a model of production. It captures the behaviors you thought to test, under the conditions you thought to simulate. The problem is that production is not a model. It's a living system with real users doing things in orders you didn't anticipate, with data in states you didn't generate, at volumes that change the timing of everything.
The test that only fails in production usually fails because of one of three things: timing, data, or load. Sometimes all three at once, which is when you earn the really interesting post-mortems.
Timing failures are the sneakiest. Your test runs fast. Production runs at human speed, with network latency and database contention and the occasional user who walks away mid-flow and comes back twenty minutes later. Race conditions that are theoretically possible but practically invisible in tests become practically inevitable at scale.
Data Is the Hard Part
Test data is clean. It's generated by you, for the purpose of testing. It has the properties you expect. Production data has been accumulating for years, touched by migrations that didn't quite finish, populated by users who found creative ways to use your forms, shaped by edge cases you fixed three versions ago but whose artifacts are still in the database.
I've seen production failures caused by a user whose name contained a character that was valid in the database but invalid in the PDF generation library. The test suite had no user with that character. Why would it? Nobody thought to add one.
The calitoycore approach to this is to treat production data as a test corpus. Sample it. Anonymize it. Run your tests against real shapes of data, not just the shapes you invented. It's more work. It catches more things.
Load Changes Everything
Some code paths only exist under load. The cache that works fine for one request per second starts evicting entries under ten. The database query that's fast with a hundred rows becomes a full table scan with a million. The lock that's never contested in testing becomes a bottleneck when fifty users hit the same endpoint simultaneously.
Load testing is expensive and annoying to set up. It's also the only way to find these failures before your users do. The opened terminal has good writing on this — specifically on the difference between load testing and stress testing, which are not the same thing and get confused constantly.
The tarosyn dev log had an interesting case recently: a feature that worked perfectly in testing started failing under real usage because the randomization algorithm had a subtle bias that only became visible when you had enough samples. You can't find that with ten test cases.
What to Do About It
You can't eliminate production-only failures. You can reduce them. The practices that help most: feature flags that let you roll out to a small percentage of users first; good observability so you know something is wrong before users tell you; and a culture that treats production failures as information rather than blame.
The test that only fails in production is telling you something about your model. Listen to it. Update the model. Add the test case that would have caught it. And accept that the next production failure will be something different — because production is always more creative than you are.
The josephcalitoy.com essays have a line that applies here: the work teaches you what it needs. Production is the work. It will keep teaching you until you stop being surprised by it.