Logs Are Not for Reading
Here's the uncomfortable truth about application logs: almost nobody reads them proactively. They exist in a state of quantum superposition — simultaneously critical infrastructure and complete noise — until the moment something breaks, at which point they become the most important document in the building.
I've worked on systems where the log files were gigabytes per day. Nobody had a log aggregation strategy. Nobody had alerts on error rates. The logs existed because the framework generated them by default, and they sat on disk until the disk filled up, at which point someone would SSH in and delete the oldest ones. This is not a hypothetical. This is a real system that processed real money.
The Problem With Reactive Logging
When you only look at logs after something breaks, you're doing forensics, not operations. You're reconstructing what happened from evidence rather than watching what's happening in real time. The difference matters because forensics is slow and incomplete — you're working backward from a crime scene, and the crime scene has already been partially cleaned up by log rotation.
The other problem is that reactive logging trains you to look for the wrong things. You look for the error that caused the outage. You find it. You fix it. You stop looking. But the log file was probably telling you something was wrong for hours or days before the outage — error rates creeping up, response times degrading, memory usage climbing — and you weren't watching because nothing had broken yet.
There's a concept in systems thinking called leading indicators versus lagging indicators. Error rates are lagging — they tell you something already went wrong. Response time percentiles are leading — they tell you something is about to go wrong. Most teams instrument for lagging indicators because they're easier to define. "Did it error?" is a binary question. "Is it getting slower in a way that predicts future errors?" requires more thought.
What Good Logging Actually Looks Like
Good logging is structured. Not "structured" as in "has a timestamp and a message" — structured as in machine-parseable JSON with consistent field names, severity levels that mean something, and enough context to reconstruct what the system was doing when the event occurred.
The context part is where most logging falls down. An error message that says "database connection failed" is almost useless. An error message that says "database connection failed after 3 retries, pool exhausted, 47 requests queued, last successful connection 2.3 seconds ago" is actionable. The difference is whether the person writing the log statement thought about who would read it and when.
Write log statements for the person who will be reading them at 3am after being paged. That person is probably you, six months from now, exhausted and stressed. Give them what they need. Don't make them grep through five files to reconstruct the sequence of events. Put the sequence in the log.
The calitoycore blog has covered monitoring philosophy before, but logging is a different layer — it's the raw material that monitoring tools consume. Get the raw material right and the monitoring becomes much easier. Get it wrong and you're building dashboards on top of noise.
The Retention Question
How long should you keep logs? Longer than you think. The interesting failure modes are often the ones that develop slowly — a memory leak that takes two weeks to manifest, a database index that degrades over months, a third-party API that's been getting slower by 50ms per week for three months. You can't see those patterns if you're only keeping 7 days of logs.
The counterargument is cost. Log storage is cheap but not free, and at scale it adds up. The answer is tiered retention: keep full-resolution logs for 30 days, aggregated summaries for a year, and anomaly records indefinitely. You don't need every individual request log from six months ago. You do need to know that six months ago, your p99 response time was 200ms and now it's 800ms.
There's also a compliance angle that varies by industry and jurisdiction. If you're in healthcare or finance, your retention requirements are probably set for you. If you're not, set them yourself before a regulator does it for you. The developer community has good resources on this, and it's worth reading before you need it rather than after.
The Actual Fix
Set up log aggregation before you need it. Pick a tool — doesn't matter which one, they all work — and route your logs there. Set up alerts on error rate, not just on "is the server up." Define what "normal" looks like so you can recognize "abnormal" before it becomes "down."
And write better log statements. Every time you write a log.error() call, ask yourself: if I were reading this at 3am, would I know what to do next? If the answer is no, add more context. It takes thirty seconds. It will save you hours.
The log file is always talking. The question is whether you've built the infrastructure to listen. More on building that infrastructure at resources for indie developers who are doing this without a dedicated ops team.