Backing up a vector database
The backup job has been green for four months. Then a bad migration truncates a collection, someone restores the most recent snapshot, and the service comes back up returning results that are subtly wrong — right document count, plausible latency, worse answers.
The snapshot was fine. What was missing was ever having checked that a restored index searches correctly, which is a different property from restoring without errors.
What makes this different from backing up a normal database
Three things.
The index is derived, and expensive to derive. You can restore vectors and rebuild the index from them, but that rebuild is hours at scale. So most snapshot mechanisms capture the built index too, which makes the artifact large and ties it to an engine version.
Correctness is statistical, not exact. A relational restore is verifiable by checksum. An ANN index is approximate by design — two correct builds over the same data return slightly different neighbours. There is no checksum that says “this index searches properly.” You have to query it.
The vectors are not the source of truth. They’re derived from documents, by a specific model, under a specific chunking strategy. A backup that captures vectors but not the model name and chunking parameters has captured something you can’t reproduce or extend.
What to actually back up
Four things, and most teams have the first.
1. The index snapshot. Whatever your engine’s native mechanism is. Fast to restore, engine- and often version-locked.
2. The source corpus, chunked and ready to embed. In object storage or a table, keyed by the same IDs the vector store uses. This is your true disaster recovery: from it you can rebuild everything, on any engine, with any model. It is small and cheap relative to what it protects. If you take one thing from this post, take this — see reindexing without downtime, which depends on it entirely.
3. The configuration. Collection schema, dimensionality, distance metric, index parameters, sharding layout, alias mappings. In version control, not in a console someone clicked through. A restored index with the wrong distance metric returns confident nonsense.
4. The provenance record. Embedding model name and version, chunking parameters, preprocessing steps, and the date. Without it, in eighteen months, nobody can tell you what these vectors mean.
Note what’s not on the list: the embeddings alone. They’re the biggest artifact and the least useful in isolation.
Snapshot mechanics
The specifics vary by engine, but the operational questions are the same:
Is it consistent? If writes continue during the snapshot, does the artifact represent a single point in time or a smear across the window? For most engines you either quiesce writes, use a mechanism that snapshots at a consistent point, or accept a bounded inconsistency and record the watermark so you can replay the tail.
Is it per-shard? Sharded deployments usually snapshot per shard, and the shards are not coordinated. Restoring a set of shards taken at different moments gives you a collection that never existed. Either coordinate them or record watermarks per shard and replay.
How much does it cost to take? Snapshots frequently involve copying the index, which means transient disk and IO. Taking one at peak traffic can itself be the incident.
What does it cost to store? These artifacts are large and they are mostly incompressible — float vectors have high entropy. Retention policy matters more here than with text-heavy databases.
Retention
A workable default, adjusted for how fast your corpus changes:
| Artifact | Frequency | Retain |
|---|---|---|
| Index snapshot | Daily | ~7 days |
| Index snapshot | Weekly | ~4–6 weeks |
| Source corpus | On every change | Indefinitely, versioned |
| Config + provenance | On every change | Indefinitely, in git |
The asymmetry is deliberate. Index snapshots are large, quick to become stale, and reconstructible. The corpus and the config are small, and losing them is unrecoverable.
The failure mode retention protects against isn’t hardware — it’s slow corruption. A bad ingestion job that has been quietly writing malformed chunks for three weeks is only recoverable if you kept something older than three weeks.
Verifying a restore, which is the whole point
Here is the procedure that separates a backup from a hope.
- Restore into a scratch environment. Never over the top of production. A restore that overwrites the thing you might still need is not a recovery, it’s a second incident.
- Check the count. Compare document count against what the source of truth says for that timestamp. A restore that silently landed 94% of the collection reports success.
- Run a fixed query set. Keep 50–200 representative queries with known-good expected results, in version control. Run them against the restored index.
- Compare recall against the reference. For each query, what fraction of the expected results appear in the returned top-k? You are not looking for a perfect match — approximate indexes vary. You are looking for the number to be in the range it was when the set was recorded. A sharp drop means the index restored but the parameters or the distance metric didn’t.
- Check latency. A restored index that is 10× slower usually means it came back unbuilt and is falling back to a brute-force scan.
- Spot-check filtered queries. Metadata indexes are a separate structure and can be missing while everything else looks healthy. Filters returning zero results is the tell.
- Record the elapsed time. This is your actual recovery time objective. The one in the document is a guess until you’ve done this.
Rollback: the scratch environment is disposable — that’s what makes this safe to do routinely. Tear it down and nothing was at risk.
Run this quarterly at minimum, and after any engine version upgrade. Version upgrades are the common way snapshots silently stop being restorable: the artifact format changed, and you find out during an outage.
The fixed query set
Worth building deliberately, because it’s the only instrument you have.
Choose queries that cover your real distribution — common ones, rare ones, short ones, long ones, several that exercise metadata filters, and a few known edge cases. For each, record the document IDs a healthy index returns in the top-k. Commit the file.
This set does more than validate restores. It’s the same instrument for validating a rebuild, an engine upgrade, or an embedding model change. If you build one thing this quarter, build this.
Two cautions. Refresh it when the corpus changes materially — expected IDs drift as documents are added and removed, and a stale set produces alarming false negatives. And regenerate it after an intentional model change, because the new geometry legitimately returns different neighbours; carrying the old expectations forward will make a successful migration look like a failure.
What good looks like
- Daily snapshots, verified quarterly by actual restore.
- Source corpus versioned in object storage, independent of the engine.
- Config and provenance in git.
- A committed query set with recorded expected results, refreshed alongside the corpus.
- A recovery time you have measured rather than estimated.
Then put the signals on a dashboard so you notice degradation before it becomes a restore — what to monitor covers which ones.