Bulk loading a collection that's already serving

Someone kicks off a backfill of two years of archived documents into the collection that is serving production traffic. Twenty minutes later query latency has doubled, segment count is climbing vertically, and the load is estimated to finish some time tomorrow. Cancelling it leaves the collection half-populated.

Bulk loading and steady-state writing are different operations with different resource profiles, and sending the first through the path built for the second is the mistake. This is how to separate them.

Why the live path is the wrong path

A trickle write is optimised for latency: accept the record, get it searchable soon, don’t disturb anything. A bulk load wants the opposite — throughput, with no requirement that any individual record be searchable quickly.

Pushing bulk volume through the trickle path produces three problems at once:

Segment explosion. Each flush creates a segment. A load that generates hundreds of them leaves every subsequent query fanning out across all of them until compaction catches up, which it won’t, because it is competing with the load.
Incremental insertion cost. Adding vectors one at a time to an existing graph means searching that graph for each one. Building a structure over a known set of vectors in one pass is a fundamentally cheaper operation, and the gap widens as the collection grows.
Resource contention with queries. Index building is CPU-heavy and IO-heavy. Doing it inside the process that is serving searches means your latency budget and your load throughput are drawn from the same pool.

Four strategies

Pick by how much spare capacity you have and how much freshness you can suspend.

1. Throttled trickle. Feed the load through the normal path, but rate-limited to a level you have measured as safe, and let it take as long as it takes. No new infrastructure, no cutover, no risk. The cost is duration — and duration matters, because a load that runs for days is a load that will be interrupted by something else.

Use when: the volume is modest relative to the collection, and nobody needs it by a date.

2. Off-peak batching. The same, but scheduled into windows where query traffic is low, with compaction given room between windows. More total throughput than a flat throttle for the same latency impact, because you are spending capacity that would otherwise be idle.

Use when: your traffic has a real diurnal shape and the load can be paused and resumed cleanly. Verify resumability before relying on it — an idempotent loader that can restart from a checkpoint is the requirement, and a loader that can’t is the actual blocker.

3. Build alongside, then swap. Create a second collection, load everything into it at full speed with no query traffic to protect, build the index in one pass, verify it, then move reads across. This is the strategy the rest of this post assumes, because it is the one that scales.

Use when: you can afford the storage for two copies for the duration, which is the same constraint as reindexing without downtime, and for the same reason.

4. Load into a new partition. If the collection is already partitioned, and the incoming data belongs to partitions nobody is querying yet — an archival date range, a new tenant — you can load into a fresh partition at full speed without touching the serving ones. Cheapest of the four when it applies, because there is no cutover and no second copy.

Use when: your partitioning aligns with the shape of the incoming data. Which is an argument for choosing it deliberately; see partitioning by the filter you always use.

The build-alongside procedure

Size it first. Two collections exist simultaneously, and the new one peaks above its steady state during the build. Work the arithmetic through sizing a vector index in memory before provisioning, and check disk headroom separately — the build needs scratch space and the eventual snapshot needs more.
Create the target with identical configuration. Same dimensionality, same metric, same index parameters, same payload schema. Any difference here becomes a quality change you will attribute to the data. Record the configuration of both so you can diff them.
Freeze the schema, not the data. Decide now what happens to writes arriving during the load — this is the step people get wrong. Options: dual-write to both collections, or queue them and replay onto the target before cutover. Dual-write is simpler to reason about and harder to make atomic; queue-and-replay is the reverse. Pick one explicitly.
Load with the bulk path if the engine has one. Many engines offer a mode that defers index construction until the data is in, or a way to import a prebuilt index. Use it. If not, load the data with indexing disabled or deferred, then build once.
Build the index in one pass, unthrottled. Nothing is serving from this collection, so give it everything. Watch memory against your peak estimate; a build that OOMs at 80% is covered in when an index build fails partway.
Verify before cutover, on three things. Record count matches the source. A sample of records round-trips with the payload intact. And recall against a fixed query set is at least as good as the current collection’s — run the same probe against both, compare, and do not cut over on count alone. A collection with the right number of wrong vectors passes a count check.
Catch up the delta. Replay the queued writes, or confirm dual-write has both collections converged. Re-run the count check afterwards.
Cut over reads. An alias if your engine has one, otherwise configuration in the client. Move a fraction first if you can and watch client-observed latency and result quality before moving the rest.
Keep the old collection for a full traffic cycle. Then drop it, and reclaim the storage.

Rollback: point reads back at the old collection. This is instant and it works at any point up to step 9, which is the entire reason for the strategy. Two conditions make it real: the old collection must still be receiving writes (or the queue must still be replayable onto it), and you must not have dropped it early to reclaim space. Both get forgotten under the relief of a successful cutover — write the drop date down instead of deciding it in the moment.

What to watch during the load

Query latency on the serving collection, client-measured. This is the number that says whether isolation actually worked. If it moves during a build-alongside load, the two collections are sharing a resource you thought was separate.
Segment count on the target. A bulk load into a deferred-index mode should not produce hundreds of segments; if it does, the bulk path isn’t engaged and you are running strategy 1 by accident.
Loader throughput and its trend. A rate that decays as the collection grows is the incremental insertion cost showing up, and it means your completion estimate is wrong in the bad direction.
Free disk, on both volumes. Two copies, plus build scratch, plus whatever compaction wants.

The general rule worth adopting: any load large enough that you’d want to estimate how long it takes is large enough that it should not go through the live write path. The threshold is not a size, it’s whether you had to ask the question.