When an index build fails partway
The index build has been running for hours. It gets most of the way through, resident memory crosses the limit, the process is killed, the orchestrator restarts it, and it starts the same build again from a state you cannot inspect. You are now in a loop that consumes a machine and produces nothing.
Two things matter in this order: stop the loop, then find out what the collection contains. Most teams do them in the opposite order and lose several more hours to restarts while investigating.
Stop the loop first
A crash-restart loop on an index build is worse than an outage, because it looks like progress. Each iteration burns IO and CPU, may leave partial artifacts behind, and resets whatever diagnostic state the previous attempt had accumulated.
- Prevent the restart. Scale the workload to zero, disable the automatic restart, or set the build to not resume on start — whichever your deployment offers. Do this before anything else.
- Capture the evidence before it is gone. The exit reason, the last log lines from the failed attempt, and the resident memory curve up to the kill. If the container was OOM-killed, the kernel said so somewhere; find that rather than inferring it from the timing.
- Confirm which failure this actually is. They present similarly and diverge immediately after.
| Symptom | Likely cause |
|---|---|
| Killed with no application error, memory curve rising to the limit | Out of memory |
| Application error naming a path or write failure, disk near full | Out of disk |
| Killed at a consistent elapsed time regardless of progress | A timeout or liveness probe |
| Error mentioning a specific record or field | Bad input, not a resource limit |
| Progress stalls without a crash | Not this post — see when ingest outruns indexing |
The fourth row is worth checking early even though it is the least common, because it is the only one where more resources will not help and the loop will continue forever.
What state is the collection in
This is engine-specific and you need the answer before you decide anything. The possibilities, from best to worst:
The build was into a new collection and nothing was serving from it. The good case. The partial artifact is disposable and you have lost only time.
The build was segment-by-segment and completed segments persist. Some engines index in units and keep finished units. A restart resumes rather than repeats, and the loop is actually making progress between crashes — slowly, and only if each iteration gets further than the last. Check whether it does before assuming it will.
The build was in place over a serving collection. Now you need to know whether queries are still being answered from the old structure, from a partially rebuilt one, or by brute-force scan. Run your fixed query set against it. If recall has collapsed or latency is an order of magnitude off, the collection is serving degraded results and that is your incident, separate from the build.
The build left an inconsistent artifact and the engine will not open it. Restore from the snapshot. This is the reason a snapshot before a rebuild is not optional, and the procedure for having one that works is in backing up a vector database.
Do not skip this determination. The remedy for a failed build differs entirely depending on whether anything is currently serving from the thing that failed.
Why it fits in a test and not in production
Nearly every case reduces to one of these.
You sized the steady state and paid the peak. Build scratch and, if rebuilding in place, a second copy of the index terms are additive on top of the resident footprint. This is the arithmetic in sizing a vector index in memory, and it is where the estimate that seemed generous stops being generous.
The build is not incremental in the way you assumed. Some engines construct the whole structure in one pass, holding candidate sets and partially built layers for the entire duration. Peak arrives near the end, which is exactly why the failure keeps happening at high progress and why the first 80% being fine tells you nothing.
The collection grew since the last successful build. Build cost grows with record count faster than linearly for graph structures, so the last build finishing comfortably is weak evidence. If you tracked build duration as a trend — one of the signals in what to monitor — you have the warning; if not, this is the incident that adds it.
Something else on the machine grew. The build was sized against a machine that also runs a sidecar, a log shipper, and a page cache full of somebody else’s data. The build’s own footprint did not change.
The payload is heavier than it was. A schema change that added a large stored field, or a re-chunk that raised record count without raising document count, moves the total without anyone associating the two events.
Getting the build to complete
Ordered by how quickly they work, not by how good they are.
- Give it a bigger machine for the build only. If the build runs somewhere separate from serving, this is a one-line change and the cheapest possible answer. Build there, then move the artifact if your engine supports importing one.
- Build in units. Split the data by partition, shard, or ID range and build each separately, so the peak is one unit’s peak rather than the collection’s. Available if your layout allows it, and an argument for partitioning that has nothing to do with query speed.
- Turn down the build’s own resource use. Fewer build threads, a smaller batch, a lower construction-time breadth parameter. All of these lower the peak and lengthen the build, and the construction-breadth one also costs recall — measure it against your query set rather than accepting the trade blind.
- Lower the connectivity parameter. Reduces both the structural term and the build’s working set. It costs recall, more so on hard queries, and it is a decision about the served index rather than about this build.
- Reduce bytes per vector. Compression cuts the largest term and helps the build and the steady state together — enabling vector compression on a live collection.
- Reduce the record count. Archive what is never queried. Slower to arrange, and it is the only option here that improves every future build too.
Prefer 1 and 2. They change where the build runs; the rest change what you serve, and a decision about what you serve should not be made under the pressure of a failed build.
Rollback: if the build was into a new collection, delete it and the previous one is untouched — this is the entire argument for never rebuilding in place. If it was in place, restore the snapshot taken beforehand, verify it with the query set, and only then plan the next attempt. For options 3 and 4, note the original parameter values before changing them, because a build that succeeds with quietly worse recall is the failure mode that survives the incident and reaches your users.
Before the next one
- Measure the peak build footprint once, deliberately, in a scratch environment at full scale. Record the ratio of peak to steady state. That one number turns this whole class of incident into arithmetic, and it feeds the projection in capacity headroom for a growing index.
- Alert on rebuild headroom, not on memory. Current footprint times your measured peak multiplier against the limit. You want to know that a routine rebuild will fail before you trigger one.
- Track build duration on every build. It is the leading indicator that the window is closing.
- Make the restart policy for build workloads explicit. An automatic restart is right for a serving process and wrong for a long batch job that failed on a resource limit. Setting that deliberately is what prevents the loop next time.