config that only exists live
env vars get added in production for a quick fix. no one backports them. your staging run never sees the real auth flow or the fallback path.
that mismatch stays quiet until a deploy lands on a tuesday night.
third party responses change
mocked apis return clean data. the real endpoints add latency and odd edge cases. rate limits appear only when traffic spikes.
our core notes tracked one team that lost three hours tracing a timeout that staging had never triggered.
scale hides in plain sight
your test dataset stays small. indexes work. joins finish fast. production tables carry years of rows and the query plan flips.
no one runs the explain until the alert fires.
background jobs behave differently
staging queues stay empty. production ones fill during peak hours and the worker pool gets starved. retries pile up without warning.
last years writeup showed how a single job type exposed the gap after months of clean deploys.
it felt fine until it didnt
people stop checking
once staging passes the checklist teams move on. the subtle drift never makes the ticket list. production becomes the only real test.
some findings from other stacks point to the same pattern repeating across unrelated codebases.
one more case showed up in this thread on env drift where the fix required rewriting the seed scripts entirely.
the cost stays hidden until the next incident.