Auditing a collection against your system of record
A user reports that a document they deleted last month is still being cited in answers. Someone else reports that a document they can see in the application never appears in search results. Both are true, and both mean the same thing: the set of records in the vector index is not the set of records your system of record says should be there.
Nothing detects this. Counts are close enough that nobody looks, recall probes pass because they test the documents you remembered to include, and the drift accumulates in both directions at once.
The four kinds of drift
Separate them, because they have different causes and different remedies.
| Drift | What it looks like | Consequence |
|---|---|---|
| Orphans | In the index, not in the source | Deleted documents resurface in results |
| Missing | In the source, not in the index | Documents are invisible to search |
| Stale | In both, index copy is from an older version | Answers cite superseded content |
| Duplicated | One source document, several index records | The same content crowds out other results |
Orphans are the ones with consequences beyond quality. A document deleted for a legal or privacy reason that is still returned by search has not been deleted in any sense that matters. Treat orphan detection as its own requirement, separate from the capacity argument, because its deadline comes from somewhere else.
Missing records are the least visible, because absence produces no complaint until a user specifically looks for something. They come from ingest failures that were retried and then given up on, from a bulk load that reported success at less than full coverage, or from a partial backfill after a schema change.
Stale records are the most common in practice. An update to a source document should be an update in the index, and any link in that chain that fails silently leaves the old version searchable indefinitely.
Duplicates come from re-ingesting without cleaning up, usually after an incident. They are the easiest to detect and they do real damage to result quality, because several near-identical records occupy several of your top-k slots.
Note what this post is not about: designing the pipeline so drift does not happen. That is a different job. This is about detecting the drift you already have and repairing the index side of it.
The reconciliation
You need a stable identifier shared by both sides. If your index record IDs are derived from the source document ID plus a chunk index, you have it. If your IDs are random and the source ID lives only in a payload field, you have it as long as that field is filterable — and if it is neither, fixing that is the first task, per changing a collection schema after it has data.
The job itself:
- Export the ID set from the source of truth, as of a recorded timestamp. Document IDs and a version marker — a content hash, an updated-at, or a revision number. The version marker is what makes staleness detectable, and if you do not have one, add one before doing anything else, because without it this audit finds three of the four drifts and not the common one.
- Export the ID set from the index. A scroll or scan over the collection, retrieving IDs and the version marker from the payload, in pages. Do this off-peak: a full scan of a large collection is a substantial read load and on a disk-resident index it will evict your working set from cache.
- Diff the two sets. Present in index only: orphan. Present in source only: missing. Present in both with different version markers: stale. Present more than once in the index for one source ID and one chunk index: duplicate.
- Record the four counts and the timestamp, and keep the history. The absolute numbers are less informative than the trend, and the trend is what tells you whether your pipeline is leaking steadily or leaked once during a known incident.
- Sample and read a handful of each category by hand before acting. This is where you find out that half your “orphans” are a chunk-ID convention that changed, or that the “missing” records are from a source type that is deliberately excluded. Acting on the counts without reading the examples is how an audit deletes something it should not have.
Expect the first run to produce alarming numbers and an explanation for most of them. That is normal and it is why step 5 exists.
Repairing each category
Orphans: delete them. Batch the deletes, align them to a partition boundary if the orphans cluster into one, and reclaim afterwards — deleting is not freeing, per why deleting vectors doesn’t free memory. Snapshot first, because an orphan list built from a bad export is a list of live records.
Missing: re-ingest them. From your source corpus, on the bulk path if the count is large — bulk loading a collection that’s already serving. Verify by re-running the diff for just those IDs rather than trusting the loader’s report, which is what reported success the first time.
Stale: re-ingest, and expect tombstones. An update is a delete plus an insert on most engines, so a large staleness repair generates churn proportional to its size and its footprint consequence arrives later, at reclamation time. If staleness is a standing problem rather than a one-off, plan a rebuild cadence instead of repeated repairs.
Duplicates: delete the extras, keeping one deterministically. Keep the newest by version marker, or the lowest ID — either is fine, as long as the rule is written down so a second run of the audit does not oscillate between them.
Rollback for all four: the snapshot you took before starting. Reclamation is not reversible, so hold that snapshot longer than your normal retention — a mistaken orphan deletion is discovered by a user weeks later rather than by a metric that afternoon. For the re-ingest categories, the rollback matters less because nothing was destroyed; the risk there is footprint, so check headroom against your projection in capacity headroom for a growing index before adding a large missing set back.
Turning it into a standing check
A quarterly manual audit is worth running. A cheap continuous one is worth more.
Sampled reconciliation, continuously. Take a random sample of source IDs on a schedule and check each one’s presence and version in the index, and a random sample of index IDs and check each against the source. Two small numbers — sampled orphan rate and sampled staleness rate — track the same thing as the full audit at a fraction of the cost, and unlike the full audit you can afford to run them hourly.
Chart both rates. A step change points at a specific date and therefore at a specific deploy or incident, which is most of the investigation. A steady climb points at a pipeline path that fails quietly.
Add them to the dashboard alongside the recall probe in what to monitor. The recall probe answers “is search working on the documents I know about.” These answer “does the index contain the right documents at all,” and they are the missing half.
Ticket rather than page on both, unless orphans have a compliance deadline attached — in which case page on the orphan rate specifically and say why in the alert.
Run a full audit after every incident that touched ingest, and after every bulk load, backfill, or migration. Those are the events that create drift, and the audit is cheapest to interpret while you still remember what happened.
The framing that gets this prioritised: your recall probe measures whether the search is good. This measures whether the corpus is right. A perfect search over the wrong set of documents scores well on every metric you currently have.