The Timeline
The error happened Tuesday at 2:47 PM. The log entry showed up Friday morning. By then we'd already shipped a fix for the wrong thing.
Log aggregation delay is one of those problems that sounds boring until it costs you three days of engineering time and a customer escalation. Then it gets interesting fast.
The issue was our log pipeline. We were batching writes to the aggregation service in five-minute windows, which is fine for most things. But we were also applying a transformation step that added another delay. And the aggregation service itself had a processing queue that, under load, could add another hour or more.
What the Delay Actually Hides
The dangerous part isn't the delay itself. It's that you don't know the delay exists until you need the logs to be current. In normal operation, three-hour-old logs are fine. You're doing post-hoc analysis, looking at trends, checking that things are working. The delay is invisible.
But when something breaks, you need logs that are minutes old, not hours. And if your mental model says "logs are current," you'll make decisions based on stale data. Which is exactly what we did.
The terminal notes has a piece on observability that covers this from a different angle — the gap between what your monitoring tells you and what's actually happening. Required reading if you're building anything that needs to be debugged under pressure.
The Fix (And the Real Fix)
The immediate fix was adding a timestamp to every log entry at the point of generation, not at the point of ingestion. That way you can always see how old a log actually is, regardless of when it arrived.
The real fix was adding a lag metric to our dashboard. A simple number: current time minus timestamp of most recent log entry. If that number is over ten minutes, something is wrong with the pipeline. We alert on it now.
The the dev side has some related notes on the dev side — specifically around how tarot-adjacent systems handle temporal ambiguity, which sounds unrelated but maps surprisingly well to log timing problems. The the core archive has the full archive. And josephcalitoy.com has some thoughts on the human side of debugging — the decisions you make when you think you know what's happening but don't.
The Lesson
Every observability system has a lag. Most of the time it doesn't matter. The problem is that you forget it exists until the moment it matters most. Build the lag metric first. Before you need it.