Catching an embedding model change nobody announced

Result quality dropped on a Tuesday. No deploy touched the vector store, the collection is intact, latency and memory are normal, and the recall probe is the only thing that noticed. Somewhere upstream, the model producing your query embeddings started producing slightly different vectors — a provider updated a hosted endpoint, a library pinned to a floating version resolved differently, or a service was rebuilt from a Dockerfile that pulls the latest weights.

Your index holds vectors from the old model. Your queries are now from the new one. The dimensionality matches, so nothing errors, and every comparison you make is between two coordinate systems that no longer agree.

Why this is the quietest failure a vector store has

The geometry changed but the shape did not. Same dimensionality, same value ranges, plausible distances. There is no validation you could add at the API boundary that would catch it, because nothing about the vector is malformed.

Partial migration is worse than total. If only query embeddings changed, every query is degraded uniformly. If the ingest path picked up the new model while the existing corpus stayed on the old one, your collection now contains two coordinate systems and results are ranked against each other across both — the mixed-geometry problem described in reindexing without downtime. Newly ingested documents will look either mysteriously dominant or mysteriously absent depending on which side the query sits on.

The magnitude of the damage is unpredictable. A minor version change may be nearly harmless; a retrained model with the same output dimensionality can be severe. You cannot infer the impact from the version number, only measure it.

Every infrastructure signal stays flat. This is the canonical case for why a vector database needs a quality signal rather than only resource signals, and it is why the probe in what to monitor is not optional.

Confirming it is this

The probe told you quality dropped. Several things cause that. This sequence distinguishes them quickly.

1. Check whether recall dropped for everything or for a subset. Uniform degradation across your whole query set points at the query side — the model, or a parameter change. Degradation concentrated in recently added documents points at the ingest side, meaning your corpus is now mixed.
2. Re-embed one query from your fixed set and compare the vector against a stored copy. This is the definitive test, and it requires that you stored query vectors alongside the expected results when you built the set. If you did, the comparison takes a minute. If you did not, add that now — it is a small change that converts this diagnosis from hours to minutes.
3. Ask the model service what version it is serving. Compare against the version recorded in your provenance record. Read it from the running service rather than from your configuration; the discrepancy between those two is the whole failure.
4. Check for a floating version anywhere in the chain. An unpinned library, a latest image tag, a hosted endpoint without a version suffix, a model file fetched at build time. Any one of these is sufficient, and the answer is often more than one.
5. Rule out the alternatives if steps 2 to 4 come back clean: a search parameter changed, an index was rebuilt with different settings, the deleted fraction climbed, or a metadata index broke. Those are the diagnosis order in when your vector database gets slow applied to quality rather than latency.

The response

You have two decisions: which model to standardise on, and how to get there.

Standardise back onto the old model if you can. Pin the version explicitly, redeploy the embedding service, and confirm with step 2 that the vectors match again. If the corpus was not contaminated, this is the entire fix and it takes minutes. It is also almost always the right choice in the moment, because it restores a known-good state without a rebuild — adopting the new model deliberately is a separate, planned piece of work.

If the old version is genuinely unavailable — a hosted endpoint that retired it — you are doing a reindex, now, unplanned. Use the blue-green strategy from reindexing without downtime rather than rebuilding in place, and accept that until it completes your results are degraded by an amount you have measured. Communicate the measured number rather than “some degradation,” because the measured number is what lets someone else decide whether to degrade gracefully or shed the feature.

If the corpus is mixed, find the boundary. The records written after the change carry the new geometry, and if your records have an ingestion timestamp you can identify them exactly. Re-embed that slice back onto the standardised model — a much smaller job than a full rebuild, and the reason to keep ingestion timestamps on every record even though nothing queries them.

Rollback: pinning the version back is itself the rollback, and it is instant. The case that needs a plan is the unplanned reindex: it needs the double footprint, and you are undertaking it without having chosen the moment. Check headroom against your projection in capacity headroom for a growing index before starting, and if the footprint does not fit, the honest sequence is to shed or archive first rather than to begin a rebuild that will fail partway — when an index build fails partway.

Making it impossible to happen quietly

1. Pin the embedding model version explicitly, everywhere. No floating tags, no unpinned libraries, no version-less endpoints. This is the fix; everything else on this list is detection for when the fix is imperfect.
2. Record the model version in every record’s payload. Then a mixed corpus is a query rather than an investigation, and the audit in auditing a collection against your system of record can count the distinct versions present as a standing check. This is the single highest-value habit in this post.
3. Store query vectors with your fixed query set, so step 2 of the diagnosis is a comparison rather than a reconstruction.
4. Add a canary embedding check to your synthetic monitoring. Embed one fixed string on a schedule and compare the resulting vector against a stored reference. Alert on any difference. This catches the change at the moment it happens, before result quality moves enough to notice, and it is a handful of lines.
5. Alert on the count of distinct model versions in the collection. It should be one. Two is an incident even if quality has not visibly moved yet.
6. Treat the embedding service as part of the database for change-management purposes. Its version changes belong in the same change log, with the same review, as an engine upgrade — see upgrading a vector database engine. Organisationally this is usually the hard part, because the embedding service is frequently owned by a different team that has no reason to think of a version bump as a database migration.

The general principle: an index built with one model and queried with another is a corrupt index that passes every integrity check you have. The only defence is provenance recorded on the data and a probe that measures quality directly, and both are cheap compared to the afternoon you will otherwise spend proving that the vector store is fine.