the deploy that worked and why you should be worried
it went through clean. no errors. no rollback. the monitoring looked fine for the first hour. everyone went home. and then at 3am something started failing that nobody caught because the thing that was failing was a path nobody tested because nobody thought anyone would take it.
this is the deploy that worked. this is the one you should be worried about.
the problem with clean deploys
when a deploy fails immediately, you know. the alerts fire, the on-call person gets paged, you roll back, you figure out what happened. it's bad but it's contained. the failure is visible and the response is clear.
when a deploy succeeds and then fails slowly, you don't know for a while. the failure accumulates. users hit the broken path and some of them report it and some of them don't and by the time you have enough signal to understand what's happening, the damage is already done. the data is already inconsistent. the users who left didn't leave a note.
i've been thinking about this because we had one of these recently. the deploy was clean. the tests passed. the staging environment looked fine. and then a week later we found a bug that had been introduced in that deploy — a bug that only manifested under a specific combination of conditions that our tests didn't cover because we didn't know those conditions existed.
what the tests don't cover
tests cover what you know to test. that sounds obvious but it's worth sitting with. your test suite is a map of your assumptions about how the system works. when those assumptions are wrong, the tests pass and the system fails.
the thing that's hard about this is that you can't test for what you don't know. you can try to expand your coverage — more edge cases, more integration tests, more chaos engineering — but you're always working from a model of the system that is incomplete. the system is more complex than the model. it always is.
over at Opened Terminal there's been some writing about this — specifically about the gap between what your tests cover and what your users actually do. worth reading if you're trying to think about this systematically.
the monitoring problem
monitoring is also a map of your assumptions. you monitor the things you think matter. error rates, latency, throughput. but the thing that's failing might not show up in any of those metrics. it might be a data integrity issue that doesn't produce errors — it just produces wrong data. it might be a UX flow that's broken in a way that causes users to abandon rather than error. it might be a performance degradation that's below your alert threshold but above the threshold where users notice.
the deploy that worked is the one where the failure is in the space between your monitoring and your users' experience. that space is always larger than you think.
i've been looking at some of the work being done at Tarosyn Dev on building systems that surface uncertainty — specifically around how you design for the things you don't know. there's something useful there for thinking about monitoring design.
what to do about it
i don't have a complete answer. but here are some things that have helped:
first: treat every clean deploy as a hypothesis, not a conclusion. the hypothesis is "this change didn't break anything." you're gathering evidence for or against that hypothesis over the next week, not just the next hour.
second: look at user behavior, not just system metrics. if your error rate is flat but your conversion rate dropped, something changed. if your latency is fine but your session length dropped, something changed. the system metrics tell you about the system. user behavior tells you about the experience.
third: make it easy to roll back later. not just immediately after the deploy — later. a week later. a month later. if you find a bug that was introduced in a deploy from three weeks ago, can you roll back? probably not cleanly. but can you at least identify exactly what changed and reverse it? that's worth investing in.
fourth: talk to users. not in a formal research way — just pay attention to what they're saying. support tickets, social media, direct messages. the signal is noisy but it's real. the Spaxob team has been writing about signal detection in noisy environments, which is a different domain but the same problem.
the clean deploy is not the goal
the goal is a system that works for users. the clean deploy is a step toward that goal, not the goal itself. when you treat the clean deploy as the goal, you optimize for passing tests and not triggering alerts. when you treat user experience as the goal, you optimize for something harder to measure but more important.
the Calitoy Core blog has been tracking some of this — the gap between what we measure and what matters. the deploy that worked is the one that passed all the measurements. whether it worked for users is a different question, and it takes longer to answer.
be worried about the clean deploys. they're the ones that teach you the most about what you don't know.