Upgrading a vector database engine

The engine upgrade goes cleanly. The pods come up, health checks pass, latency is normal, and three weeks later somebody notices that answer quality has been slightly worse since a date nobody connects to the deploy. The upgrade did not break availability. It changed a default, or a construction detail, or how a parameter is interpreted, and your results moved.

This is the characteristic failure mode of upgrading a vector database, and it is why the procedure here differs from upgrading a stateless service. The check that matters is not “does it start” — it is “does it still return the same neighbours.”

Read the notes for four specific things

Release notes for a database this young often bury the operationally significant items among features. Look for these, in this order.

Index format changes. If the on-disk or in-memory structure changed, the upgrade either migrates it on start — which can take as long as a build and is not always resumable — or requires a rebuild. Both are fine, and being surprised by either during a maintenance window is not. Find out which, and how long it takes on a collection your size.

Default parameter changes. A changed default for connectivity, construction breadth, search breadth, or segment merge policy will alter recall or latency on any collection that relies on the default rather than setting the value explicitly. This is the commonest cause of the silent quality change. It is also the argument for setting every parameter you care about explicitly in your configuration, even where the default is what you want, so that an upgrade cannot move it.

Snapshot compatibility, in both directions. Forward compatibility — can the new version read old snapshots — decides whether your existing backups survive the upgrade. Backward compatibility — can the old version read new snapshots — decides whether you can roll back at all once the new version has written anything. The second is the one nobody checks, and it is what turns a rollback into a restore from a pre-upgrade snapshot plus a replay.

Distance metric or scoring changes. Rare and severe. A change in how a score is computed or normalised changes ranking, thresholds, and any score cutoff in your application. If your service filters on a similarity score, this is the item to search the notes for hardest.

If the notes do not say, assume the worst case for each and test for it. “The notes did not mention it” is not evidence.

Before you touch production

Snapshot, and verify the snapshot restores on the current version. Not a new snapshot you trust — one you have actually restored. The full check is in backing up a vector database, and this is the moment its absence hurts most.
Confirm the pre-upgrade snapshot is readable by the new version too. Restore it into a scratch environment running the target version. If it is not readable, your recovery path from a failed upgrade is “restore onto the old version,” which means keeping the old version deployable — check that your images and manifests still allow that.
Record the baseline. Recall against the fixed query set, p50/p95/p99 for the real query mix, resident footprint, and segment count. All from the same window. This is the comparison set and there is no substitute for having taken it beforehand.
Write down every index parameter currently in effect, including the ones you are getting by default. Read them from the running system rather than from your configuration file, because those two disagree more often than anyone expects, and the difference between them is exactly what an upgrade can change.
Upgrade a scratch environment restored from production data. Not a small test collection — the format migration and the build behaviour are size-dependent, and a fast migration on a tiny collection tells you nothing about the window you need.
Re-measure everything from step 3 on the upgraded scratch environment. Recall first. If recall moved, find the parameter that moved with it before proceeding; do not accept “approximate search varies” as the explanation for a shift that reproduces.
Time the migration and multiply. Whatever the format migration took in scratch, your window needs to be comfortably longer, and you need to know whether it can be interrupted.

The upgrade

For a replicated deployment, one replica at a time.

Freeze other changes. No parameter tuning, no schema changes, no bulk loads in the same window. You want exactly one variable.
Reduce or pause ingest if you can. A format migration competing with writes is slower and, on some engines, changes what happens if it is interrupted.
Take the immediately-pre-upgrade snapshot. Even though you have one from step 1. This is the restore point.
Upgrade one replica and leave it out of the read path. Let its migration complete. Watch its footprint — a new format can be larger, and a migration can peak above steady state the way a build does, with the failure mode in when an index build fails partway.
Query it directly with the fixed query set before it serves anything. This is the gate. Compare recall and latency against the baseline, not against the other replicas.
Give it a fraction of reads and compare client-observed latency against the replicas still on the old version. Running both versions side by side under real traffic is the best comparison you will ever get, and it exists only during a rolling upgrade — use it.
Watch for a cold start. A freshly upgraded replica has an empty page cache and possibly a rebuilt structure, so its first queries are slow for reasons unrelated to the upgrade. Warm it before judging it — see warming a vector index after a restart.
Proceed replica by replica, re-checking recall after each rather than only at the end.
Hold the mixed state as briefly as you reasonably can. Different versions may produce different results for the same query, and while that is usually within the noise of approximate search, it makes any quality investigation during the window ambiguous.
Re-verify snapshots after the upgrade. Take one on the new version and restore it. Your backup procedure is unproven on this version until you have.

Rollback: if the format did not change, redeploy the previous version — the data is compatible and this is a normal rollback. If the format did change, the new version has rewritten the collection and the old version cannot read it, so the rollback is: restore the step 3 snapshot onto the old version, then replay writes from the window. That is much slower, which is why steps 2 and 3 exist and why you want ingest paused. Know which of these two situations you are in before you start, and state it in the change record.

Afterwards

Keep the recall probe running through the upgrade window and for a week after. A quality change that only shows on certain query types will not appear in a single post-upgrade check.
Diff the effective parameters against the list from step 4. Any that moved, move back explicitly or accept deliberately.
Re-measure the peak build multiplier if the format changed, and update the projection in capacity headroom for a growing index. A new format with a different footprint changes every capacity number you have.
Note the version next to your provenance record. When somebody restores a snapshot in eighteen months, the engine version that wrote it is the first thing they will need.

The rule that prevents most of this: pin every parameter explicitly, and treat an upgrade as a quality change until measured otherwise. An engine upgrade on a vector database is not infrastructure maintenance. It is a change to how your search ranks results, and it deserves the same verification as a model change.