Failing over to a vector store replica

The primary node goes away. Traffic moves to the replica, which is exactly what the replica is for, and query latency is an order of magnitude worse than it was. Nothing is broken. The replica has never served a query, so nothing about it is warm, and on some deployments the structure it holds is not even the same structure the primary was serving from.

Replication on a vector database covers less than people assume, and the parts it does not cover are the parts that decide whether a failover is a blip or an incident. This is about finding out which before the failover is involuntary.

What is actually being replicated

Ask your engine’s documentation this question specifically, because there are three answers and they fail differently.

What is replicated	What the replica has to do on promotion	Where it hurts
The built index itself	Serve	Nothing structural — only cache warmth
The write stream, each replica indexing independently	Nothing, if it has kept up	Replicas can diverge in structure and recall
The raw records, index built on demand	Build	Promotion takes as long as a build

The middle row is the interesting one and it is common. If each replica constructs its own index from the same writes, the resulting structures are not identical — graph construction depends on insertion order and on concurrency, so two replicas fed the same records in slightly different order hold genuinely different graphs. Both are correct. Both return slightly different neighbours. Your recall is therefore a property of which replica answered, which is fine until you are debugging a quality complaint and cannot reproduce it.

The bottom row is the one that turns a failover into an outage, and it is worth confirming you are not in it before you rely on a replica for anything.

The gaps replication does not close

Warmth. A replica that has served nothing has a cold page cache and, if it is disk-resident, nothing useful in it. Its first minutes of traffic will be slow for reasons that have nothing to do with the failover. This is the most common cause of a failover looking like a failure, and it is covered in warming a vector index after a restart.

Indexing lag on the replica. If replicas index independently, a replica can be behind on indexing even while it is current on writes — it has the records, they are not searchable yet. That is the condition in when ingest outruns indexing, and a replica is where it hides, because nobody queries the replica so nobody notices.

Compaction state. Segment counts diverge between replicas that compact independently. A replica with a much higher segment count will be slower and may have measurably worse recall, per compaction, and when to force it. Segment count is a per-replica metric and averaging it across the set hides exactly this.

Configuration drift. Parameters set by hand on the primary during an incident, six months ago, and never applied to the replicas. This is why every index parameter belongs in version-controlled configuration rather than in a runtime change nobody recorded.

Corruption and bad writes. Replication copies mistakes faithfully. A retention job that deleted the wrong month is replicated within seconds. Replicas are an availability mechanism, not a backup — the backup is in backing up a vector database, and the two are not substitutes.

Deciding what a failover should cost you

Write these down before rehearsing, because a rehearsal without expectations just produces a number nobody can judge.

How much latency degradation is acceptable, and for how long. A warm-up period is not a failure if you have decided it is not.
How much recall degradation is acceptable. If replicas hold independently built structures, some movement is expected. Establish the range.
How much write loss is acceptable. Asynchronous replication means a window of writes exists only on the failed primary. For a vector store this is usually recoverable — the records are still in your source corpus and can be re-sent — which makes the tolerance higher than for a transactional database. Confirm that the pipeline can replay from a watermark rather than assuming it.
Whether partial results are acceptable. In a sharded deployment, a lost shard may yield a result set silently missing a fraction of the corpus. Decide whether you want that or an error, and then find out which one you actually get.

Rehearsing it

Do this on a schedule, in production, deliberately. A failover path that has only ever run during an incident has not been tested.

Verify the replica is current before you start. Replication lag, indexing lag, and record count against the primary. If any of the three is behind, you have found something more useful than the rehearsal was going to tell you.
Compare the replica against the primary on the fixed query set, querying each directly. Recall and latency. This is the single most informative check in this post and almost nobody has run it — it tells you whether your replica would serve the results you expect, which is the whole question.
Diff the effective configuration between primary and replicas, read from the running systems.
Warm the replica, then fail over, then measure. Do the warm-up separately from the failover so you can attribute the latency you see.
Fail over cold, separately. This is the realistic case, because an involuntary failover does not wait for you to warm anything. Record how long the degradation lasts and how bad it gets. That number is your actual failover cost and it belongs in the runbook.
Watch client-observed latency, not engine-reported latency, throughout. The engine’s own timing will not include connection re-establishment, retries, or client-side queuing, and those are a real part of what users experience during a failover.
Check the write path. Confirm writes are landing on the new primary and that nothing upstream is still pointed at the old one, holding a stale connection.
Fail back, and measure that too. Failing back is a second failover with a cold former primary, and teams that rehearse the first frequently discover the second is worse.

Rollback: fail back to the original primary once it is healthy and warm. The condition that makes this safe is that the original primary must not have accepted writes after the failover — a split brain where both nodes took writes leaves you reconciling two divergent collections, and for a vector store the reconciliation is usually “rebuild from the source corpus,” which is the expensive path. Confirm the fencing behaviour of your deployment before rehearsing, not after.

What to monitor per replica

The signals in what to monitor need a per-replica breakdown, and these in particular stop being useful as aggregates:

Replication lag and indexing lag, separately. A replica current on one and behind on the other is the state that surprises people.
Segment count per replica. Divergence here predicts a slow failover.
Resident footprint per replica. A replica that is closer to its limit than the primary will be the first to be OOM-killed, and it will happen when it takes traffic.
The recall probe, run against each replica rather than through the load balancer. Through the balancer you are sampling whichever node answered, which averages away the one that is wrong.
Time since last failover rehearsal. A dashboard number, deliberately. It is the only signal that degrades purely through inattention.

The summary worth carrying: a replica is not a spare until it has served traffic and you have measured what it returns. Until then it is a copy of your data with unknown behaviour, which is a useful thing to have and not the thing you think you have.