Query timeouts and retry storms

Latency drifts up past the client’s timeout. The client gives up and retries. The original query is still running inside the engine, consuming a worker, because a client disconnecting does not always cancel server-side work. Now there are two queries where there was one, the queue grows, more requests cross the timeout, and each of them retries. The database has not failed; it has been converted into a failure by the retry policy in front of it.

This is the fastest way a vector database goes from slow to down, and every part of it is configuration you own.

Why vector search amplifies this worse than most

Three properties make an ANN query an unusually bad thing to retry.

The work is not cancelled by a disconnect. A graph traversal or a multi-partition probe runs to completion inside the engine unless the engine explicitly checks for cancellation. Whether yours does is the first thing to find out, because it decides whether a retry adds load or replaces it. Assume it does not until you have confirmed otherwise.

The cost per query is variable and partly caller-controlled. Search breadth, top-k, filter selectivity, and result payload size all move the cost of a single query substantially. So the same retry policy that is harmless against a uniform workload can be ruinous against one where a minority of queries are much more expensive — and the expensive ones are exactly the ones that time out and get retried.

The failure mode being retried is usually saturation, not a fault. Retries are for transient faults — a dropped connection, a node restarting. When the cause is that the index no longer fits in memory or compaction has fallen behind, every retry makes the cause worse. A retry policy tuned for network faults applied to a capacity problem is an amplifier.

Choosing the timeout

The number is derived, not chosen. Three inputs.

Your latency budget. What the calling service can wait for and still meet its own commitment. This is the ceiling and it comes from outside the database.

Your measured tail. p99, and p99.9 if you have it, for your real query mix under realistic load. A timeout below your normal p99.9 fires routinely on healthy queries, which trains everyone to ignore timeout alerts.

The gap between them. If your measured tail is close to your budget, you do not have a timeout problem, you have a latency problem, and setting a tighter timeout will convert slow responses into errors without improving anything. Fix the latency first — when your vector database gets slow has the diagnosis order.

Set the timeout on both sides. A client-side timeout stops the caller waiting. A server-side timeout, where the engine supports one, stops the work. Only the second actually protects the database, and it is the one usually left unset. Where both exist, the server-side one should be slightly shorter, so that work stops before the client stops waiting — otherwise you get abandoned work running behind every timeout, which is the whole problem above.

Retries that shed load instead of adding it

If you retry, four properties are not optional.

Cap the attempts, low. More than one retry against a saturating backend is rarely useful, because the second retry arrives when conditions are worse than the first found them.

Back off, and jitter. Immediate retries synchronise. Exponential backoff with randomised jitter is the standard answer and it matters more here than usual, because a synchronised burst of expensive traversals is the worst possible arrival pattern for an index under memory pressure.

Budget retries as a fraction of traffic, not per request. A per-request retry limit still permits every request to retry simultaneously. A retry budget — a cap on retries as a proportion of total requests over a window — is what actually prevents the storm, because it goes to zero exactly when everything is failing, which is when retrying is most harmful and least useful.

Do not retry what will not succeed. Distinguish error classes: a timeout under load should usually not be retried, a connection reset during a rolling restart should. If your client treats every failure identically, it is retrying the one case where retrying is destructive.

Add a circuit breaker. When the failure rate over a window crosses a threshold, stop sending and fail fast for a cooling period. This is the only mechanism in the list that gives the database room to recover, because everything else still sends some traffic. It is also what makes a failover useful — see failing over to a vector store replica — since a breaker that trips can route elsewhere.

Degrading instead of failing

A vector search has something most database queries do not: a quality dial. That makes graceful degradation genuinely available, and it is underused.

Under load, you can:

• Reduce search breadth. Fewer nodes explored, faster query, lower recall. A worse answer delivered is usually better than no answer, and unlike a timeout it is a decision you made rather than one that happened to you.
• Reduce top-k, if the caller asked for more results than it displays.
• Skip the rerank stage, where you have one.
• Serve from a cheaper replica or a cached result, for repeated queries.

The important part is that these are explicit modes with explicit triggers, not permanent settings quietly lowered during an incident and never restored. Lowering search breadth is the classic incident-response change that survives forever and silently costs recall — that is why it is step 5 and not step 1 in the diagnosis order.

If you implement a degraded mode, instrument it: a counter for how many queries were served degraded, and the recall probe running against both modes so you know what the degraded answer is actually worth. A degraded mode whose cost you have never measured is a quality regression waiting for a busy week.

The runbook when you are in a storm

1. Confirm it is a storm and not a slowdown. Look at request rate arriving at the database against request rate leaving the caller. If the first is a multiple of the second, you are looking at retries. This comparison takes seconds and is the whole diagnosis.
2. Check the percentile shape. A p99 that rose and then plateaued at a suspiciously round value is a timeout, not a slowdown — the same tell as in the diagnosis table.
3. Stop the amplification. Turn retries down or off at the client, or trip the breaker manually. Do this before investigating the underlying slowness; you cannot diagnose a system whose load is a function of its own latency.
4. Shed load deliberately. Reject at the edge, or switch to the degraded mode. Choosing what to drop is better than having it chosen for you.
5. Then find the underlying cause, which is now visible because the load is stable.
6. Restore retries and the breaker afterwards, and record what you changed.

Rollback: every step here is a client or edge configuration change and reverts by setting it back. Step 3 is the one to write down before changing, because “we turned retries off during the incident” and then leaving them off removes real resilience against the transient faults retries are for. Put the original values in the incident notes at the moment you change them.

Prevention

• Set a server-side timeout, not just a client one. If your engine has no server-side timeout, bound the work another way — a cap on search breadth and top-k is a proxy for it.
• Cap the caller-controlled parameters, per tenant where relevant — isolating tenants in one vector store.
• Alert on retry rate as a distinct signal, not folded into request rate. A rising retry rate is a leading indicator of everything in this post and it is invisible in a total.
• Alert on error rate by class, per the dashboard in what to monitor. Timeouts and connection errors mean different things and a single counter loses the distinction that decides your response.
• Test it. A load test that pushes past saturation with your real retry policy engaged is the only way to know whether your configuration sheds or amplifies — load testing a vector database before it matters.

The single most valuable change, if you make one: a server-side timeout slightly shorter than the client’s. It converts the worst case from unbounded abandoned work into bounded wasted work, and it costs one configuration line.