Load testing a vector database before it matters

The load test passed comfortably, and the first real traffic peak produced latency the test never showed. The test generated random query vectors against a synthetic collection, held every parameter at its default, and ran for a few minutes from a warm process. Every one of those choices removed a failure mode that production has.

A vector database load test is easy to run and easy to run meaninglessly. The difference is almost entirely in the inputs.

Why random query vectors are useless here

A randomly generated vector is not in the same region of the space as your real queries. Real queries cluster — they land near the parts of the corpus people actually ask about, and they traverse the same regions of the index repeatedly, which is precisely why caching works at all.

Consequences of testing with random vectors:

Cache behaviour is wrong in the pessimistic direction for hit rate and the optimistic direction for the conclusion, because a uniform miss pattern across the whole structure looks like steady load rather than the hot-and-cold pattern that produces tail latency.
The traversal is a different shape. A query in a dense region explores differently from one in a sparse region, and random vectors in high dimensions tend to land nowhere near anything.
Result sets are different. Random queries return whatever is least far away, which affects payload size and serialization cost.
Filter interaction is absent, unless you also synthesised realistic filters, which nobody does.

Use real queries. Sample them from your logs. If you cannot log query text, log the embedded vectors or their hashes; if you cannot do that, sample real documents from the corpus and use their vectors as queries, which is imperfect but lands you in the right regions. Anything that puts you in the real distribution beats a generator.

What the test needs to include

The real query mix, in the real proportions. Filtered and unfiltered, cheap and expensive, the long-tail queries and the ones that repeat constantly. If a tenth of your traffic is an expensive query shape, a test with none of it is testing a different service. Sample rather than curate — curated mixes drift toward the queries someone found interesting.

A collection at the size you are testing for. Not today’s size. Insertion and traversal cost both grow with record count, so a test against a small collection answers a question you are not asking. If you cannot get real data at scale, at least get real vectors at scale.

Concurrent writes. Production is never read-only. Indexing and compaction compete with queries for the same resources, and a read-only load test excludes the contention that causes most real latency problems. Run the test with your actual write rate applied, and with a bulk load running for at least part of it.

A cold start. Begin measuring before the process is warm, and record the curve. Steady-state performance is one number; how long you are bad for after a restart is another, and it is the one that shows up during a deploy — warming a vector index after a restart.

Your real retry policy and timeouts. This is the most commonly omitted element and it inverts the result of the test. A load test with retries disabled will find the saturation point; a load test with your production retry policy engaged will find out whether crossing that point degrades or collapses — query timeouts and retry storms.

Long enough to hit background work. Compaction, snapshots, and segment merges are periodic. A test shorter than the interval between them never overlaps with them, and that overlap is where your sawtooth latency comes from.

What to measure

Beyond the obvious throughput and latency numbers:

The full latency distribution, at p50, p95, p99 and p99.9. Never the mean. The interesting behaviour of an approximate index lives in the far tail, and a mean hides a change in tail shape completely.
Recall, during the load. This is the measurement almost no load test takes and it is the one that distinguishes a vector database test from a generic one. Run your fixed query set concurrently with the load and record recall as a series. Under contention some engines will return faster, worse results, and a test that only measures latency will report that as a good outcome.
Resident footprint over time, including whether it stabilises. A slow climb during a long test is either a leak or a growing structure, and both matter.
Segment count over time. If it climbs monotonically under your write rate, you have found a compaction configuration problem before it found you — compaction, and when to force it.
Indexing lag. Whether writes are becoming searchable at the rate they arrive, per when ingest outruns indexing.
Client-observed latency alongside engine-reported latency. The gap between them is queuing, and under load the gap is where the interesting behaviour is.

The shape of the run

Establish the baseline at low load, warm, and record everything above. This is the reference and nothing later means anything without it.
Ramp gradually and hold at each step. A step and hold pattern shows you where the curve bends; a single jump to the target tells you only whether you passed.
Find the saturation point deliberately. Keep ramping until latency degrades past your budget. You want this number, and you want it in a test rather than in production. It is the input to every capacity conversation you will have.
Then re-run past saturation with retries enabled. The two runs together tell you whether your system sheds load or amplifies it, and that is a different property from where it saturates.
Hold at your expected peak for a long period. Long enough for background maintenance to run at least once. This run is where the sawtooth appears.
Repeat once with a bulk load running concurrently. This is realistic, and it is the run that most often produces a surprise.
Record every parameter of the run. Collection size, index parameters, query mix source and date, write rate, hardware, engine version. A load test result without its inputs cannot be compared against the next one, which removes most of its long-term value.

Rollback: if you are testing against a scratch environment, there is nothing to roll back and that is the point — this is one of the few procedures on this site that is inherently safe. If you are testing against production, which is sometimes the only way to get realistic conditions, then the shed and abort path is the rollback: a hard cap on the generated load, a single control to stop it, and someone watching client-observed latency for real users rather than the test’s own numbers. Agree the abort threshold before starting.

What to do with the result

Turn the saturation point into a capacity number, and put it in the projection alongside the memory ceiling in capacity headroom for a growing index. Query capacity is a ceiling like the others and it is usually the one nobody has quantified.
Set the timeout from the measured tail, not from a round number.
Re-run after anything that changes the index. A compression rollout, a shard split, an engine upgrade, a new embedding model. Each invalidates the previous result, and each is a change you would like to have measured before it reaches users.
Keep the query sample fresh. Query distributions drift with the product. A sample from a year ago tests last year’s service.

The honest summary: the value of a load test on a vector database is almost entirely in whether the inputs resemble production. A rigorous methodology over synthetic vectors produces a precise measurement of something you do not operate.