The Green Build That Lied
A passing test suite is supposed to mean something. Green means go. Green means the code does what you think it does. Except sometimes it doesn't, and the tests pass anyway, and you ship something broken with full confidence.
False positives in test suites are, in some ways, worse than failing tests. A failing test is a signal. A false positive is silence — the absence of a signal where there should be one. You don't know what you don't know.
I've seen this happen in a few specific ways, and they're worth naming because they're not always obvious when you're in the middle of them.
The Mocked Dependency That Doesn't Match Reality
You mock an external service. The mock returns what you expect. The test passes. Then the real service changes its response format, and your mock doesn't know, and your tests still pass, and you find out in production.
This is the most common flavor. The mock was accurate when you wrote it. Then it became a lie, slowly, without anyone noticing. The test is testing your mock, not your code's behavior against the real thing.
Contract testing helps here. So does occasionally running integration tests against the real service in a staging environment. Neither is a complete solution. Both are better than nothing.
The Test That Asserts the Wrong Thing
This one is subtler. The test runs, makes assertions, passes. But the assertions don't actually verify the behavior you care about. They verify something adjacent to it — something that's usually correlated but isn't the same thing.
Example: you test that a function returns a non-null value. It does. But you didn't test that the value is correct. The function could be returning garbage and the test would still pass.
The fix is to be more specific in your assertions. But that requires knowing what "correct" looks like, which requires understanding the behavior deeply enough to specify it. Sometimes the act of writing a precise assertion reveals that you don't actually know what the function is supposed to do.
The Test That Never Runs
A test that's been accidentally excluded from the test runner. A test file that's not in the right directory. A test that's been marked as skipped and nobody remembers why. These tests show up in the suite, they show as passing (or not failing), and they're not testing anything.
Regular audits of your test coverage — not just line coverage, but behavioral coverage — catch this. Opened Terminal has been writing about the gap between coverage metrics and actual confidence, which is worth reading alongside this.
The broader point is that a green build is a hypothesis, not a proof. It means your tests didn't find a problem. It doesn't mean there isn't one. The Tarosyn dev blog has an interesting parallel in the context of randomness and determinism — the idea that absence of evidence isn't evidence of absence. And Spaxob covers this in the context of signal detection, which maps surprisingly well to test design.
Treat your test suite as a living document that needs maintenance. The tests that pass without being read are the ones to worry about.