Changing a collection schema after it has data

Product wants to filter results by a field that is already stored in the payload but was never declared filterable. The change looks like one line of configuration. Depending on your engine it is either exactly that, or it is a full reload of every record, or it is a rebuild of the entire index — and you will not know which until you have checked, by which point you may already have given someone a date.

Schema changes on a vector collection sort into three tiers of cost. Working out which tier you are in is the whole job; the procedure afterwards is one you already have.

The three tiers

Tier 1 — metadata only, no data movement. Adding a payload field that nothing indexes, relaxing a validation rule, adding an alias. The existing records simply do not have the new field, and whether that is acceptable is an application question rather than a database one.

Tier 2 — a secondary structure has to be built, but the vector index is untouched. Making an existing payload field filterable, adding a sort key, changing the type of a metadata field. The vectors and their graph or cluster structure stay exactly as they are; a separate index over the metadata is constructed. This is much cheaper than a vector rebuild and it is not free — it needs memory that your sizing did not account for, and building it competes with serving.

Tier 3 — the vector index must be rebuilt. Anything that changes the vectors or how they are compared: dimensionality, distance metric, compression settings, and on most engines the core index parameters. Also, frequently, adding a partition or shard key, because the layout is part of how the collection was created.

The awkward middle case is tier 2 pretending to be tier 1. On several engines a field must be declared filterable before the records are written, and declaring it afterwards indexes only records written from that point on. Queries then return results filtered against a partial index — no error, no warning, and a result set that is silently missing everything older than the change. This is the failure mode to check for specifically, because it is the one that looks like it worked.

Finding out which tier you are in

Do not reason about it from the shape of the change. Test it.

1. Create a scratch collection with your current schema and load a modest sample of real records — real vectors, real payloads. A synthetic sample will not exercise type edge cases in your metadata.
2. Apply the change. Note whether the engine accepts it at all, whether it returns immediately or works for a while, and whether resident footprint moves.
3. Query for the oldest records through the new field. This is the test that catches the partial index. If records written before the change are not returned by a filter on the new field, you are in tier 2 with a backfill requirement, whatever the documentation implied.
4. Query for the newest records too, so that a total failure is distinguishable from a partial one.
5. Measure the footprint delta, and scale it to your production record count. A metadata index over a high-cardinality field is not small, and this is the number that decides whether the change fits.
6. Check whether it is reversible. Drop the new field or the new index and see whether the collection returns to its previous state and footprint. Several engines will not release the memory, and knowing that in advance changes how you plan the rollout.

That whole sequence is under an hour and it replaces an argument with an answer.

Rolling out a tier 2 change

The cheap-looking one, which needs the most care because it is the one people do casually.

1. Size the new structure using the measured delta from step 5, and confirm you have that headroom against your projection in capacity headroom for a growing index. A metadata index built into a collection that was already close to its ceiling is an OOM with a schema change in the change log.
2. Take a snapshot. Cheap insurance, and the change is being applied to a serving collection.
3. Apply it off-peak, and watch query latency while the structure builds, not afterwards. You are adding a CPU and IO consumer to a serving process.
4. Backfill if the test said you must. Rewriting every record’s payload so the new field is indexed is an ingest job at the scale of your whole collection, and it belongs on the bulk path rather than the live one — bulk loading a collection that’s already serving. Expect it to generate segments and tombstones, because on most engines a payload update is a delete plus an insert.
5. Verify coverage, not completion. Count records matching the new filter and compare against what the source of truth says should match. A backfill that covered most of the collection reports success, and the gap is invisible from inside the database — the reconciliation is in auditing a collection against your system of record.
6. Re-run the recall probe with the new filter applied. Add filtered queries on the new field to the fixed query set permanently. An unfiltered probe cannot detect a broken metadata index.
7. Expect to need reclamation afterwards if step 4 rewrote records. The tombstones are real and the footprint will not fall on its own — see why deleting vectors doesn’t free memory.

Rollback: drop the new index or field, per what step 6 of the testing sequence told you about reversibility. If the engine does not release the memory, the honest rollback is a restart, and if the change cannot be dropped at all, the rollback is the snapshot from step 2 plus a replay of the window. Establish which of the three you have before applying the change, and write it in the change record — this is exactly the kind of change that gets applied without a rollback plan because it looked small.

Rolling out a tier 3 change

It is a rebuild. Use the strategy you already have: build a new collection with the new schema, load from your source corpus, verify, cut over, keep the old one for a full traffic cycle. The strategies and their memory costs are in reindexing without downtime and nothing about a schema change alters them.

Two additions specific to schema:

• Diff the two schemas explicitly before loading, field by field, and record the diff. A rebuild that accidentally changed a second thing is the hardest quality regression to attribute, because you will be looking for a cause in the change you knew about.
• If dimensionality or the metric changed, the results genuinely change and comparing recall against the old collection’s expected IDs is meaningless. Regenerate the expected results after the cutover rather than treating the difference as a failure.

Preventing the next one

• Declare fields filterable when you create the collection, even the ones you do not filter on yet. The cost is the metadata index; the saving is not needing a backfill later. Where the cost is material, at least declare the ones that plausibly become filters — tenant, source, date, status.
• Keep the schema in version control and apply it from there, so nobody discovers the effective schema differs from the intended one during an incident.
• Treat schema as part of the provenance record alongside the model name and chunking parameters, per backing up a vector database. A restored snapshot with an undocumented schema is a puzzle.
• Ask for the filter requirements when the corpus is designed, not when the feature is scoped. The distance between those two moments is the difference between one line of configuration and a fortnight.