The Test That Passes Locally and Fails Everywhere Else

The Test That Passes Locally and Fails Everywhere Else

The test passes on your machine. It fails in CI. You add some logging, push again — it passes in CI. It fails in staging. You add more logging. It passes in staging. It fails in production. At this point you've spent four hours on a test that takes 200 milliseconds to run, and you're starting to wonder if the test is haunted.

The test is not haunted. But it is telling you something important about your assumptions.

What Environment-Specific Failures Actually Mean

When a test passes locally and fails elsewhere, the instinct is to blame the environment. The CI container is different. The staging database has different data. Production has different load. All of that is true, and none of it is the real problem.

The real problem is that your test has hidden dependencies on things you didn't know you were depending on. The local environment satisfies those dependencies by accident — because of how your machine is configured, what's in your local database, what time zone your system is set to, what version of a library you happen to have installed.

The test isn't failing in other environments. The test is revealing that it was never actually testing what you thought it was testing.

The Most Common Culprits

Time zones are responsible for more environment-specific test failures than almost anything else. A test that creates a record at 11:30 PM local time and then queries for records created "today" will fail in CI if CI runs in UTC and your local machine runs in Pacific time. The test isn't wrong about the logic. It's wrong about what "today" means.

File system assumptions are second. Tests that write to a temp directory, assume a specific path separator, or depend on case-insensitive file matching will fail on Linux CI after passing on macOS. The calitoycore codebase has a few of these — they surface every time someone runs the test suite on a new machine.

Database state is third. Tests that assume a clean database, or that assume specific seed data exists, or that assume a particular ordering of results without an explicit ORDER BY — these are time bombs. They pass until they don't, and when they fail, the failure is almost impossible to reproduce reliably.

There's good writing on this at openedterminal, particularly around the philosophy of what a test is actually for. The short version: a test that only passes under specific conditions isn't testing your code. It's testing your environment.

The Logging Trap

When a test fails intermittently, the natural response is to add logging. This is almost always a mistake, for a specific reason: the act of adding logging changes the timing of the code. A race condition that manifests without logging may not manifest with it. A test that fails because of a timing issue may pass once you've added enough print statements to slow things down.

You think you've fixed it. You haven't. You've just changed the conditions enough that the failure doesn't appear in the same place. It'll be back.

The better approach is to make the test deterministic. If it's a timing issue, mock the clock. If it's a database state issue, set up and tear down explicitly. If it's a file system issue, use a test-specific temp directory that you control. The goal is to eliminate the hidden dependencies, not to work around them.

What the Failure Is Actually Telling You

An environment-specific test failure is a design smell. It means your code has implicit dependencies on the environment that aren't expressed in the code itself. Those dependencies exist whether or not the test reveals them — the test is just making them visible.

The tarosyn dev log has a good example of this from a few months ago — a card draw function that produced different results depending on whether the system clock was in DST or not. The test caught it in CI. The fix wasn't to fix the test; it was to make the function explicitly accept a clock parameter instead of reading the system time directly.

That's the pattern. When a test fails in a specific environment, ask: what is this environment providing that my local environment provides differently? Then make that dependency explicit. Pass it in. Mock it in tests. Stop relying on the environment to provide it by accident.

The Test That Can't Be Fixed

Sometimes you find a test that genuinely can't be made deterministic without rewriting the code it's testing. This is useful information. It means the code has a design problem — it's doing too much, or it's coupled to things it shouldn't be coupled to, or it's making assumptions that should be parameters.

The test failure is a gift. It's showing you where the design needs work. The maker mindset here is to treat the failing test not as an obstacle but as a diagnostic — it's pointing at something real, and the something real is worth fixing.

Fix the code. The test will follow.