When ingest outruns indexing

The writer reports success on every upsert. Nothing is erroring. But queries have got slower each day this week, and a document ingested an hour ago sometimes appears in results and sometimes doesn’t. The index is behind the data, and the engine is hiding it from you by accepting writes it hasn’t finished processing.

Indexing lag is distinct from a compaction backlog and it has a different fix. Compaction is about merging structures that already exist; this is about records that are stored but not yet in any searchable structure.

What happens to an unindexed record

Engines handle the gap between “written” and “indexed” in one of three ways, and you need to know which yours does before anything else in this post is actionable.

Behaviour	Symptom you’d see
Brute-force scan over the unindexed buffer	Latency rises with the size of the backlog
Not searchable until indexed	Recent documents intermittently missing
Indexed synchronously on write	Write latency rises; ingest throughput caps out

The first is the most common and the most confusing, because the degradation is smooth and proportional. A small unindexed buffer scanned exhaustively is genuinely cheap, which is why the design works. A large one is a linear scan bolted onto every query, and it grows at exactly the rate your ingest exceeds your indexing throughput.

The second is worse to debug and better to have: at least the failure is visible as a correctness complaint rather than a slow drift. If your users report that a document they just added isn’t findable, and it becomes findable later, stop looking at the retrieval pipeline and go look at indexing lag.

Finding the number

You want two figures: the size of the unindexed backlog, and its rate of change.

Where to look:

An explicit metric. Names vary — unindexed count, pending records, buffer size, indexing queue depth. If your engine has one, this is a five-minute job.
Segment state. Some engines distinguish segments that are built from segments that are accumulating. The size of the accumulating one is your backlog.
Inferred from behaviour. Insert a marker document, then poll for it with a query that should match it uniquely. Time to first match is your visibility lag. This is crude, it only samples one point, and it is far better than nothing — it also happens to be exactly the check you want in a synthetic monitor.

Chart the backlog, not just the lag. A lag of a few seconds that is stable is fine. A lag of a few seconds that has been doubling daily is the same incident three days earlier, and this is the whole value of watching it.

Why indexing can’t keep up

Indexing is not proportional to ingest. Adding a record to a graph structure means searching the existing structure to find its neighbours, then updating their lists. That work grows with the size of the index, not with the size of the batch. A pipeline that kept up comfortably at a few million vectors can fall behind at twenty million with no change in write rate at all. This is the most common cause and the least intuitive one.

It’s throttled, or single-threaded, or both. Background index building is usually resource-capped so it doesn’t starve queries. Check what the cap is, and check how many threads it is allowed. The default was chosen without knowledge of your ingest rate.

It’s competing with compaction. Both want IO and CPU. When both are behind they slow each other down, and the system can settle into a state where neither ever catches up. Look at both before concluding which is the cause.

Writes arrive in a shape that defeats batching. Many small upserts, each triggering an index operation, cost far more than the same records in batches. If your writer sends one record per request because that was easiest, this is a cheap and large win.

Updates are counted twice. An update to an existing record is generally a delete plus an insert. A pipeline that re-upserts unchanged records — which is what a naive full re-sync does — generates enormous indexing work for no change in content. Check whether your writer is re-sending records whose content hasn’t changed; deduplicating upstream is usually the entire fix.

The runbook

Confirm it is indexing lag and not compaction. Both show as rising latency. Indexing lag shows a growing unindexed buffer or a growing visibility delay; compaction shows a growing segment count. Fix the one that is actually moving.
Stop the bleeding: reduce the write rate. If a backfill is running, pause it. If a re-sync is running, pause it. The backlog cannot drain while it is being added to faster than it clears, and every other step is slower than this one.
Check for re-upserts of unchanged records. If a large fraction of writes are no-ops in content terms, the fastest fix is upstream and it is permanent.
Raise the indexing resource budget, if the engine allows it and there is headroom. Watch query latency while you do — you are moving capacity from serving to maintenance, deliberately, and you want to know how much that costs before you leave it that way.
Batch the writes. Larger batches, fewer requests. This usually requires a writer change, so it is not the incident fix, but it belongs in the follow-up.
If the backlog is very large, consider a rebuild instead of a drain. Building an index over the full dataset in one pass is often faster than incrementally indexing a large backlog into an existing structure, for the same reason incremental insertion is expensive. The procedure is reindexing without downtime; the decision hinges on whether the backlog is a significant fraction of the collection.

Rollback: every step above is a rate or resource change and reverts by setting it back. The one that needs care is step 4 — if raising the indexing budget degrades query latency past your budget, put it back and accept a longer drain, because a slow index is better than a slow service. Note the original values before changing them; “we raised something and can’t remember what it was” is the usual state of a system a week after an incident.

Prevention

Alert on backlog size and visibility lag, both as trends. These are the leading indicators; query latency is the lagging one.
Include a freshness probe in your synthetic monitoring. Write a marker, query for it, record the delay. It catches this class of problem before users do, and it is the only check that covers the “not searchable until indexed” case at all.
Route bulk work away from the live write path, as in bulk loading a collection that’s already serving.
Re-test indexing throughput at your projected size, not your current one. Because insertion cost grows with index size, capacity planning for writes has to be done against the future collection. Fold it into the headroom review rather than treating write capacity as fixed.

The thing to internalise: a vector database that accepts a write has not necessarily made it searchable, and the gap between those two states is a queue with its own capacity limits. Treat it like any other queue — measure depth, measure drain rate, alert on the trend, and apply backpressure before it becomes unbounded.