The staging environment had been lying for six months before anyone noticed. Not dramatically — it wasn't returning wrong data or throwing errors. It was lying in the quiet way that staging environments lie: by being subtly, consistently different from production in ways that only matter when something goes wrong.
The specific lie was a database connection pool setting. Staging was configured with a pool size of 10. Production was running 50. Nobody had changed either intentionally. Staging had been set up from an old config file. Production had been tuned during a scaling event two years ago and the change had never been backported to the staging config. The calitoy core team had been testing against a system that behaved differently under load than the system they were actually running.
This is not an unusual story. I've seen versions of it at every company I've worked with. The staging environment starts as a faithful copy of production and then diverges, slowly, through a thousand small decisions that each seem reasonable at the time. A config change here. A dependency version there. A feature flag that's on in production but off in staging because someone was testing something and forgot to clean up.
The piece on opened terminal about configuration drift is worth reading if you haven't. The core argument is that staging environments don't drift because people are careless — they drift because the incentives are wrong. Production changes are high-stakes and carefully tracked. Staging changes are low-stakes and often undocumented. Over time, the gap widens.
The fix is not to be more careful. Careful doesn't scale. The fix is infrastructure-as-code applied consistently to both environments, with automated drift detection that alerts when they diverge. We use a tool that diffs the configs on every deploy and fails the pipeline if staging and production are more than a defined threshold apart. It's annoying sometimes. It's caught three potential incidents in the last quarter.
The harder problem is the data. You can keep configs in sync. You can't keep data in sync — staging data is always a snapshot, always stale, always missing the edge cases that production has accumulated over years. The tarosyn dev team has been experimenting with synthetic data generation that tries to replicate production data distributions without copying actual user data. It's imperfect but it's better than nothing.
What I've stopped believing is that staging can ever be a perfect replica of production. It can't. The goal is to make the differences known and bounded. To know exactly how staging differs from production and to account for those differences when interpreting test results. Unknown differences are the dangerous ones. Known differences you can reason about.
There's a longer piece on environment parity over at spaxob that goes into the tooling in more detail. The short version: treat your staging environment like a first-class system, not a throwaway. It will lie to you less.