Running a vector index from disk

Your index no longer fits in memory, the instance size above yours costs more than the project has, and someone has suggested “just put it on disk.” That is a real option on several engines. It is also the change most likely to turn a latency problem into a latency incident, because vector search assumes memory-speed random access and disk does not provide it.

This post is about deciding whether disk residency is viable for your workload, and how to roll it out so you find out cheaply rather than during peak traffic.

What “on disk” actually means here

Three different things get called the same thing, and they fail differently.

Arrangement	What lives in RAM	Where it hurts
Fully resident	Everything: structure and vectors	Nothing — this is the baseline
Memory-mapped, OS-managed	Whatever the page cache happens to hold	Unpredictably: cold pages fault mid-query
Deliberately tiered	The structure; vectors read on demand	Predictably: a bounded number of reads per query

The middle row is the dangerous one, and it is also the default on several engines when you simply point them at a larger dataset than the machine has RAM for. Nothing errors. The index builds. The first queries are fine because they touch hot pages. Then a query walks into a cold region and takes two orders of magnitude longer than its neighbours, and your p99 develops a long tail that no dashboard explains.

The bottom row is what you want if you are doing this on purpose. The distinguishing property is that the traversal structure stays in memory and only the vector data is fetched. A graph traversal is a sequence of dependent lookups — each hop determines the next — so if the hops themselves require disk reads, the reads serialize and the latency is the sum of them. If the hops are in RAM and only the final distance computations need data from disk, those reads can be issued together.

That distinction is the whole decision. Before you enable anything, find out which of the three your engine gives you when configured this way. If the documentation does not say clearly whether the index structure remains resident, assume it does not and test accordingly.

Whether your storage can take it

Vector search on disk is a random-read workload with small request sizes and no useful readahead. That is the worst case for any storage device, and the gap between device classes is much wider here than for the sequential throughput numbers on a spec sheet.

What matters, in order:

Random-read latency at your queue depth, not sequential throughput. A device that streams fast can still serialize small random reads.
Whether the storage is local or network-attached. Network block storage adds a round trip to every read that misses cache, and it is the round trip, not the bandwidth, that ends up in your p99.
Whether the device is shared. Co-tenanted or throttled storage will hand you latency variability you cannot fix from inside the database.
Free space. Disk-resident indexes still need headroom for compaction and snapshots, and now the data volume is large enough that “we’ll add a disk later” is a maintenance window.

Measure it directly rather than reasoning from the instance type. Run a random-read benchmark against the actual volume, at a request size close to what one vector occupies, at a concurrency close to your query rate times the reads-per-query you expect. If that number is bad, disk residency will not work for you and no configuration will rescue it.

Compression is usually the better answer first

Before moving anything to disk, check whether you can make the thing fit. Reducing the bytes per vector component keeps the entire structure in memory and pays a recall cost you can measure and bound; moving to disk keeps recall identical and pays a latency cost that varies per query and is much harder to bound.

Bounded quality loss is easier to operate than unbounded tail latency. Try compression first — see enabling vector compression on a live collection for the rollout — and reach for disk only when the compressed index still doesn’t fit, or when your recall requirement genuinely won’t tolerate compression.

The two also combine, and the combination is often the right shape: compressed vectors resident in memory for the traversal, full-precision vectors on disk, read only for the final candidates. If your engine supports that, it is strictly better than either alone, because the disk reads are few and happen once per query rather than at every hop.

The rollout

Establish the baseline you will be judged against. Record p50/p95/p99 for your real query mix while fully resident. Also record recall against your query set, because you are about to change something that should not move recall, and confirming that is how you know you configured tiering rather than something else.
Build a disk-resident copy alongside the resident one. Same data, same parameters, different collection. Do not convert the serving index in place; you want both to exist so the comparison is real and the rollback is instant.
Replay production queries against the copy with the page cache in a realistic state. This is the step people skip, and it is the only one that matters. A freshly built index has its data warm in the page cache, so the first measurement flatters it enormously. Drop caches or restart the process, then replay, and read the distribution rather than the average.
Look at the tail specifically. The failure mode is not a uniform slowdown; it is a small fraction of queries becoming very slow. Compare p99 and p99.9 between the two collections. If p50 barely moved and p99 tripled, you have the memory-mapped case, not the tiered case.
Shadow it, then shift a fraction of live reads. Send a small percentage of real traffic to the disk-resident collection and watch client-observed latency, not just engine-reported latency.
Keep the resident index until a full traffic cycle has passed. Weekly patterns produce query mixes your replay didn’t contain.

Rollback: point reads back at the resident collection. This is why step 2 insists on a second collection rather than an in-place conversion — an in-place conversion’s rollback is a rebuild, which is exactly the expensive operation you were trying to avoid. Keep the resident copy until you are certain, and accept the double footprint for that period as the cost of a cheap way back.

What to monitor afterwards

Disk residency changes which signals matter, and most vector-database dashboards do not have them. Add:

Page-cache hit ratio, or the engine’s equivalent cache statistic. This is now your primary leading indicator. A gradual decline means the working set is outgrowing the cache and the tail is about to get worse.
Read amplification per query — storage reads divided by query count. If this climbs, the traversal is touching more of the structure than it was, usually because segment count grew.
Storage-level latency, separately from query latency. When they move together the storage is the cause; when query latency moves alone it is something else, and you have saved yourself an investigation.
p99.9, not just p99. The whole risk profile of this change lives in the far tail.

Segment count deserves particular attention once you are on disk, because the fan-out multiplier that was merely annoying in memory becomes a multiplier on disk reads. Keep compaction healthy, and treat a rising segment count as urgent rather than untidy.

The honest summary: disk residency trades a hard limit for a soft one. Fully resident, you find out you have a problem when the process is killed. On disk, you find out when a fraction of your users are slow, and the fraction grows quietly. The second is easier to survive and much easier to ignore, so the monitoring above is not optional — it is the thing that replaces the OOM as your signal.