This content originally appeared on Level Up Coding – Medium and was authored by Anas Anjaria
A real-world DB outage — why replicas aren’t the silver bullet — and how to dig deeper for lasting fixes.
Real-world examples often teach us more than any theory. In my day-to-day work, I’ve learned the most from incidents that forced me to think deeper.
Recently, I spotted one on LinkedIn that struck me as the perfect teaching moment. I’ll share it here anonymously — not to criticize, but to explore how to investigate properly, weigh tradeoffs, and design long-term fixes.
TL;DR
A disk failure caused downtime in a standalone database setup. The team’s fix? Add a replica.
But here’s the problem:
Replicas reduce downtime, but don’t fix the root cause.
Clusters add cost, complexity, and replication lag.
The better approach → recover fast, then investigate and fix the actual failure mode for a future-proof solution.
Context: Database Outage in Production
The situation: a team had a standalone database setup. One day, their only primary node went down. Result → complete outage.
In response, they decided to move from standalone to a cluster setup by adding a replica node. That was their solution.
When I asked about the root cause, they said it was a disk failure.
This immediately raised questions for me.
Does adding a replica actually solve a disk failure?
Open Questions for You
Before diving into my thoughts, let’s pause:
- Does adding a replica really address the root cause?
- Is it a long-term, future-proof fix?
- If it were your system, how would you have approached it?
My Concerns With the Replica Solution
Don’t Jump Into Solutions Without Root Cause Analysis
Rule #1 — Never jump straight into a solution without understanding the root cause.
Yes, adding a replica reduces the blast radius of downtime. But the disk failure risk hasn’t gone anywhere — it’s still lurking. The same issue could occur again, requiring manual intervention. That’s a band-aid, not a cure.
Complexity: From Standalone to Cluster
Distributed systems aren’t free. They’re inherently complex, and as engineers we should simplify whenever possible, not complicate.
By moving from a standalone node to a cluster, you invite new challenges:
Higher cost: With rising infrastructure costs (especially in the AI era), most companies prioritize cost efficiency. Adding nodes without addressing the actual failure may not be justifiable.
Management overhead: One node is simple. Clusters mean more moving parts — even with cloud-managed services, upgrades and failovers become trickier.
Eventual consistency: Classic cluster problem. Replicas can fall behind, serving stale reads. Your business logic must account for this. I’ve written about it here.
Workload distribution: Typically, replicas exist not just for HA, but to share workload. If your replica is only for failover, is the added complexity really worth it?
So, while the fix “works,” it creates a different set of long-term problems. Sometimes, simplicity (a single node, well-monitored) is more reliable than an over-engineered cluster.
How I Would Have Approached It
I’ve made plenty of mistakes in my own journey, so I don’t claim this is the only way. But here’s how I’d approach the problem.
Step 1: Recover Fast
Get the system healthy again. That’s a no-brainer.
Step 2: Investigate the Root Cause
Once the fire is out, dig deeper:
- Why did the disk fail?
- Was it hardware-specific, or an issue with the cloud provider?
- Could it be mitigated with monitoring, redundancy, or storage configuration?
The point: your long-term fix should align with the actual failure mode.
A Similar Example from My Experience — WAL Segment Corruption
We faced a similar case in production. One of our nodes encountered corrupted WAL (Write Ahead Log) segments.
The quick fix? Restarting the node.
The long-term fix? We discovered it was tied to ZFS compression bugs. So, we moved WALs out of ZFS compression.
That was future-proof. We didn’t just mask the problem. We removed the failure mode itself.
Lessons Learned
Don’t jump into “obvious” solutions — solve the root cause first.
Replicas = availability, not resilience against every failure.
Complexity adds cost; keep systems simple unless justified.
Investigations pay long-term dividends — quick fixes don’t.
Conclusion
Every outage is a chance to learn. Adding replicas isn’t wrong — but it’s not always right either.
The real value comes from pausing, investigating, and designing fixes that reduce future risk, not just today’s downtime.
Question for you: Have you ever seen replicas added as a “quick fix” when the real root cause was elsewhere?
I write actionable, experience-based articles on backend development — no fluff.
Find more of my work, connect on LinkedIn, or explore upcoming content: all-in-one
Database Outage: Is Adding a Replica Always the Right Fix? was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding – Medium and was authored by Anas Anjaria