The N+1 query problem is one of those things that looks fine in development, looks fine in staging, and then detonates in production when you have actual data. You've seen it. You've probably caused it. I've caused it more times than I'd like to admit, and each time I'm briefly convinced it's a new and unique problem before I recognize the shape of it.
Here's the pattern: you have a list of things. For each thing, you make a database query to get related data. In development, your list has 10 items. You make 11 queries — one for the list, ten for the related data. That's fine. In production, your list has 10,000 items. You make 10,001 queries. Your database is now on fire.
The reason it hides so well is that ORMs make it invisible. You write something like posts.map { |p| p.author.name } and it looks like you're just accessing a property. You're not. You're making a database call for every post. The ORM is helpfully doing the work you didn't ask it to do, in the worst possible way.
Why Detection Is Hard
The problem with N+1 queries is that they're not errors. They're not slow in development. They don't show up in your test suite unless you're specifically looking for them. They're just quietly wrong, waiting for scale to make them visible.
The dev culture and builder mindset conversation around this is interesting — there's a real tension between writing code that's readable and writing code that's efficient. The ORM pattern that causes N+1 is often the most readable version of the code. The fix — eager loading, joins, batching — is often less readable. You're trading clarity for performance, and in development, clarity wins every time because performance isn't a problem yet.
The tools that help: query logging in development (turn it on, always), bullet_gem for Rails, Django Debug Toolbar for Django, similar tools for most frameworks. These make the invisible visible. You see 47 queries where you expected 2 and you know something is wrong.
The Fix Is Usually Simple
Once you've found it, the fix is usually straightforward. Eager load the associations you know you'll need. Use a join when you're filtering on related data. Batch your queries when you're processing large sets. The hard part is finding it, not fixing it.
There's a the personal cost of technical debt angle here too. N+1 queries are a form of technical debt that's particularly insidious because they feel like they're working. The feature ships. The tests pass. The code review doesn't catch it because nobody ran the query log. And then six months later, when the data has grown, you're debugging a production incident that traces back to a line of code that looked completely reasonable when it was written.
I've been thinking about this in the context of tarot and probability engines — systems that need to make many small queries efficiently. The card draw logic is a good example: if you're pulling card data for a spread, you want one query, not one per card. The pattern is the same everywhere.
The discipline is: before you ship anything that touches a list, look at the query log. Count the queries. If the count scales with the list size, you have an N+1. Fix it before it ships. It's much easier to fix in development than to debug in production at 2am when the database is at 100% CPU.
More on systems and building at Calitoy Core.