Splitting a collection across shards

The collection no longer fits comfortably on one machine, the next instance size up buys one doubling and then you are here again, and rebuilding it takes long enough that you have quietly stopped doing it. Sharding is the answer to all three. It is also weeks of work, which is why it has to be started at the point where you can still see the ceiling rather than at the point where you have hit it.

Everything below assumes you have read the capacity projection in capacity headroom for a growing index and know roughly when your ceilings arrive. Sharding is the mitigation you start at the Watch threshold, not the Act one.

What sharding fixes, and what it does not

It fixes the per-machine ceiling. Total memory does not fall — it rises slightly, because each shard carries its own structural overhead and the fixed per-index costs are paid more times. What changes is that no single machine holds all of it.

It caps the blast radius of a rebuild. This is the benefit that matters most and is mentioned least. Rebuilding one shard needs one shard’s worth of peak headroom rather than the collection’s, so the second-copy term in sizing a vector index in memory gets divided by your shard count. A collection you had lost the ability to rebuild becomes rebuildable again, one piece at a time.

It gives you a unit for every other operation. Compression rollouts, disk-residency trials, engine upgrades, and reindexes all become per-shard changes with per-shard rollbacks. Most of the procedures on this site get safer once sharding exists.

It does not fix query cost, and may worsen it. A query that must consider all data now fans out to every shard, gathers candidates from each, and merges. The merge is cheap; the fan-out is not free, and your latency becomes the slowest shard’s latency rather than the average. That is a real regression and it is the thing to measure.

It does not fix recall, and can change it. Each shard is searched with its own budget and returns its own candidates. Splitting the same data across shards means each traversal sees a thinner slice of the neighbourhood, in exactly the way a compaction backlog does — the mechanism is described in compaction, and when to force it. Raising per-shard search breadth compensates, at a latency cost. Measure recall before and after and expect to retune.

Choosing the shard key

Two families, and the choice is more consequential than the shard count.

Random or hash-distributed. Records are spread evenly with no relationship to their content or metadata. Every query fans out to every shard. You get perfectly even sizing and no routing logic, and you pay the fan-out on every request.

Choose it when queries have no natural boundary, or when the size distribution of every candidate semantic key is badly skewed.

Semantic — tenant, region, date, source. Records that belong together live together, and a query carrying that key touches one shard. Fan-out disappears for those queries, and per-shard operations become per-tenant or per-period operations, which is operationally excellent.

Choose it when nearly all queries carry the key. The trap is skew: real tenants are not evenly sized, so one shard ends up holding most of the data and you have a single-machine ceiling again with extra complexity. Check the distribution before committing — the same check as in partitioning by the filter you always use, and if you already partition on a key, that key is your first candidate for a shard key.

The hybrid that usually wins: semantic sharding with the large keys split further, so the largest tenants occupy dedicated shards and the long tail shares. More routing logic, and it is the only arrangement that gets both even sizing and single-shard queries.

Whatever you choose, write down how you will change it, because you will. A shard key you cannot re-shard is a decision you have made permanently on the strength of this quarter’s data.

Deciding the count

Not a formula, but four constraints that bracket it.

Each shard’s peak build footprint must fit its machine, with the multiplier from your own measurement. This sets the minimum.
Fan-out cost sets the maximum. More shards means more parallel work per unfiltered query and a wider distribution of per-shard latencies feeding your tail. Past some count, adding shards makes queries worse faster than it makes rebuilds better.
Growth headroom. Pick a count that still fits when the collection is as large as your projection says it will be at your planning horizon, so you are not re-sharding within the year.
Re-sharding cost. Splitting a shard in two is a rebuild of that shard. A count that divides cleanly when you next double is worth more than a count that is optimal today.

The migration

Baseline everything you will be judged on. Recall against your fixed query set, p50/p95/p99 for the real query mix, and the current footprint. Recall especially, because sharding can move it and you need to know by how much.
Check the size distribution of the candidate key against your actual data, not your assumptions about it. This is the step that changes the plan.
Provision the target and build alongside. A sharded collection next to the unsharded one, same dimensionality, metric, and index parameters. Load from your source corpus.
Verify per shard. Record counts per shard against what the routing rule says each should hold. A routing bug passes a total count check and fails silently on the queries that land in the wrong place.
Measure recall on the sharded collection and retune search breadth if it dropped. Do this before any traffic moves. Re-measure latency after retuning, because you have just bought recall with latency and need to know the price.
Measure the fan-out cost specifically. Split your query mix into queries that carry the shard key and queries that do not, and compare each against the baseline separately. The unfiltered slice is where the regression lives, and an aggregate comparison hides it.
Confirm what happens when one shard is unavailable. Some engines return partial results, some error. You need to know which, before it happens — and if it is partial results, whether your service can tell. See failing over to a vector store replica.
Shadow, then shift a fraction of reads, watching client-observed latency at p99 and p99.9 rather than the mean.
Keep the unsharded collection for a full traffic cycle, then drop it and reclaim.

Rollback: point reads back at the unsharded collection. It requires the unsharded collection to still exist and still be current, which means dual-writing or a replayable queue for the whole migration period — and that period is long for sharding, which makes the rollback expensive to keep and therefore the thing most likely to be quietly abandoned. Decide explicitly how long you will carry it, and note that once dropped, your rollback is a full rebuild rather than a configuration change.

What changes on the dashboard

Sharding replaces one set of signals with a distribution of them, and a dashboard that averages across shards will hide every problem sharding introduces. Add:

Per-shard footprint and record count. The imbalance chart. This is now your primary capacity view and the aggregate is nearly useless.
Per-shard query latency, not just the merged figure. Your tail is the slowest shard, so an average across shards tells you nothing about what users experience.
Shards touched per query. Rising means queries are arriving without the key and you are losing the benefit.
Partial-result rate, if your engine can serve one. A shard silently dropping out of a fan-out degrades quality with no error and no latency change, and nothing else detects it.
Per-shard build duration, which is now your rebuild lead time.

The signals in what to monitor all still apply — they just need a per-shard breakdown behind each of them.

The honest summary: sharding converts a hard ceiling into an operational surface. You gain the ability to grow and to rebuild, and you take on imbalance, fan-out, and partial failure as permanent concerns. That trade is almost always worth making, and it is much easier to make before the ceiling than after.