Replication Failover

Replication failover promotes a suitable replica to become the writable primary when the current primary is unavailable. It restores write availability, but the process must account for replication lag, data loss risk, split-brain prevention, and application reconnection.

A replicated database typically has a primary that handles writes and one or more replicas that receive copied changes. During failover, an eligible replica is selected and promoted to primary so clients can resume writes. The exact mechanism depends on the database and replication architecture.

A key concern is whether the replica contains every committed change from the old primary. With synchronous replication, acknowledged data may exist on multiple nodes, reducing loss during promotion. With asynchronous replication, the promoted replica can be behind, so transactions committed immediately before the failure may be absent.

After promotion, clients must stop sending writes to the old primary and discover the new primary. Applications commonly use a database endpoint, service discovery mechanism, proxy, or driver configuration rather than permanently embedding one server address. Connection pools may also need to discard stale connections and reconnect.

A robust failover design prevents split brain, where two database nodes independently believe they are primary. Leader election, fencing, consensus-based coordination, or provider-managed orchestration can be used to establish a single authoritative writer. Recovery of the old primary also requires care: it may need to be rebuilt or resynchronized before returning as a replica.

For a Java backend engineer, failover is therefore both a database and application-availability concern. The service should tolerate transient connection failures, retry only operations that are safe to retry, and avoid assuming that a previously opened database connection remains valid after a topology change.

Key points

• Failover promotes a replica to replace an unavailable primary.
• Asynchronous replication can cause acknowledged data to be missing after promotion.
• Clients must discover the new primary and refresh stale database connections.
• Split brain must be prevented so only one node accepts authoritative writes.
• The old primary usually needs resynchronization before rejoining as a replica.

Interview questions

What happens during database replication failover?

The system detects that the current primary is unavailable and selects an eligible replica. That replica is promoted so it can accept writes. Applications must then connect to the new primary rather than continuing to use the failed node. The exact promotion and discovery mechanism depends on the database architecture.

Can failover cause data loss?

Yes, depending on the replication mode and how much data had reached the replica selected for promotion. With asynchronous replication, the replica may lag behind the old primary, so recently committed transactions might not have been copied before failure. Synchronous replication can reduce this risk when its configured acknowledgment guarantees require the necessary replicas to persist the change. The actual durability guarantee depends on the database's replication configuration.

How should a Java service handle a database primary changing during runtime?

The service should use a topology-aware or managed database endpoint when available rather than permanently targeting one server. Connection pools need to discard broken connections and establish new ones against the current writer. Transient failures can be retried when the operation is known to be safe, but blindly retrying writes can duplicate effects or obscure an already committed transaction. Timeouts, bounded retries, and idempotency are important parts of the design.

What is split brain, and why is it dangerous during failover?

Split brain occurs when multiple nodes independently believe they are the authoritative primary. If both accept writes, the system can produce conflicting histories that are difficult or impossible to merge safely. Failover orchestration therefore needs a mechanism that establishes one authoritative writer, such as leader election combined with fencing or provider-managed coordination. Preventing dual writers is generally more important than simply detecting that one server is down.

What should happen when the failed primary comes back after failover?

It should not automatically resume accepting writes merely because it is reachable again. Its state may be behind or may contain transactions that diverged from the newly promoted primary. The node should be reconciled or rebuilt according to the database's recovery procedure and then rejoined as a replica. Only after synchronization and health checks should it be considered eligible for future promotion.