Warming a vector index after a restart

The pod restarts, the health check goes green, the load balancer sends it traffic, and the first few minutes of queries are far slower than normal. Then it settles. Every deploy produces a latency spike that nobody has ever investigated because it always goes away, and it is the same spike that turns a node failure into a visible incident.

Something is cold. Working out which something decides whether the fix is a readiness probe, a warm-up routine, or a capacity change — and the three are not interchangeable.

Four things that can be cold, and they are not the same

The page cache. The index file is on disk and the operating system has nothing from it in memory yet. Every read that misses faults from storage. This is the dominant effect for anything memory-mapped or disk-resident, and its severity depends entirely on your storage — the arrangement matters more than the size, as in running a vector index from disk.

The index load itself. Some engines read the whole structure into their own heap at startup rather than relying on the page cache. That is a bounded, measurable startup cost — the process is not ready, and it will say so. Slower to start, faster once started, and much more predictable.

The index build. If the engine constructs rather than loads — because the format requires it, or because the restart interrupted a build — startup is a build, with the duration and the peak footprint of one. If your restarts sometimes take minutes and sometimes take hours, this is why.

Everything around the database. Connection pools, JIT warm-up in the client, DNS, and the embedding service’s own caches. It is worth ruling these out early, because they present identically and they are not fixed by anything you do to the database.

The diagnostic that separates them: watch resident memory and storage read rate during the warm-up. A rising resident footprint with heavy reads is a load or a page-cache fill. A flat footprint with heavy reads is page-cache thrash and it will not settle — that is a capacity problem, not a warm-up one. High CPU with little IO is a build.

The readiness gate is the actual fix

Most warm-up incidents are not slow warm-ups. They are traffic arriving before the node was ready, because the health check answers a question that has nothing to do with whether queries will be fast.

A liveness check should say “the process is alive.” A readiness check should say “this node will serve queries at the expected latency.” For a vector database those are separated by the entire warm-up, and the default health endpoint answers the first.

What a readiness check should actually do:

1. Confirm the index is loaded and searchable, not just that the process is listening.
2. Run a small number of representative queries and require them to complete within a threshold you set. This is the part that matters — a query that succeeds slowly should fail readiness.
3. Confirm the record count matches expectation, so a node that came up holding a partial or stale collection never receives traffic.
4. Only then report ready.

Getting this right converts the whole problem from “users see a latency spike on every deploy” to “deploys take longer,” which is a trade worth making every time.

Warming deliberately

Once readiness is gated correctly, the remaining work is making the warm-up shorter.

Replay a representative query sample. The best warm-up is your own traffic. Keep a sample of real queries — the same fixed set you use for the recall probe is a reasonable starting point, though a larger sample covers more of the structure — and run it against the node before it reports ready. This touches the regions of the index that real queries touch, which is the property no synthetic warm-up has.

Do not warm with random queries. Random vectors traverse the index in ways your traffic does not, so they fill the cache with pages you will not use and take as long as real ones. Warming with the wrong queries is close to not warming at all.

Consider reading the index file sequentially. Where the index is memory-mapped and the machine has the memory for it, a sequential read of the file pulls it into the page cache far faster than random query traffic will, because sequential IO is the one thing storage is good at. Only do this if the whole thing genuinely fits; if it does not, you are evicting as you go and achieving nothing.

Look for an engine-provided preload option. Several engines can be told to load the structure eagerly at startup rather than lazily on first access. If yours can, that is strictly better than anything you build, and it makes the cost visible at startup where it belongs.

Restart one node at a time, and wait. The rolling restart that takes down two of three replicas concurrently puts full traffic on one node that is also the only warm one. Rolling with a real wait between nodes is the difference between a slow deploy and an incident.

The rehearsal

1. Measure a cold start honestly. Restart a node out of the read path, then replay traffic against it with the page cache genuinely cold — dropped, or after a machine restart. Record the latency distribution over time until it flattens. That curve is your warm-up cost, and its length is the number your deploy process needs to respect.
2. Measure it again with your warm-up routine in place, the same way. The difference is what the routine bought.
3. Record both in the runbook. During a real incident, “this node needs several minutes and this is expected” is a valuable thing to know and an easy thing to have written down.
4. Repeat after the collection grows materially, or after a format or compression change. All three change the warm-up, and the recorded number silently becomes wrong.

Rollback: a warm-up routine and a stricter readiness probe are both configuration, and both revert by putting them back. The one that needs care is the readiness threshold — set it too tight and nodes never report ready, which is an outage caused by the fix. Deploy the stricter probe to one node first, confirm it does eventually pass, and only then apply it everywhere.

When warm-up is not the problem

Two cases where the warm-up is a symptom and warming harder will not help.

It never settles. Latency improves and then plateaus somewhere worse than it used to be, or keeps oscillating. The working set does not fit and the cache is thrashing. This is a capacity problem — capacity headroom for a growing index — and warming just moves which queries are slow.

Restarts are getting longer over time. If startup duration is trending up, the index is growing into a load or build cost that will eventually exceed whatever timeout your orchestrator enforces, at which point the node is killed mid-startup and restarted, which is the loop in when an index build fails partway. Track startup duration as a trend alongside build duration on the dashboard in what to monitor; it is one of the cheapest leading indicators available and almost nobody collects it.

The habit worth adopting: treat every deploy as a rehearsal for a failover, because mechanically it is one. If your deploys produce a visible latency spike, your failovers will produce a worse one, and the fix is the same fix.