ColBERT-Serve

A serving system that memory-maps the ColBERT index, uses a sparse first stage (SPLADEv2) to pick candidates, and re-ranks them with ColBERT under a hybrid score. Cuts RAM by 90% and lets late-interaction retrieval run on inexpensive servers under many concurrent queries. ECIR 2025, with Matei Zaharia (Databricks co-founder), Christopher Potts, et al.

2023–2025 · started at Stanford, published from Microsoft ECIR 2025 paper Code GitHub stars for ColBERT-Serve retrieval · serving · systems
>90%
less RAM for the ColBERT index
multi-stage
SPLADE candidates, ColBERT re-rank with hybrid scoring
ECIR 2025
first author

The problem. ColBERT is one of the best retrievers available, but its index stores a vector per token, so it does not fit in memory on a cheap server. Most deployments either pay for RAM or give up on late interaction.

The idea. Treat the index like a database, not a tensor. ColBERT-Serve memory-maps the compressed ColBERTv2 embeddings from disk and relies on the OS page cache, so only the pages a query touches get loaded. Memory-mapping alone makes search about 2× slower because of page misses, so a sparse first stage (SPLADEv2 running on the PISA engine) picks the top-200 candidates and ColBERT only scores those. The final ranking uses a hybrid score, a z-normalized interpolation of the SPLADE and ColBERT scores, which on MS MARCO beats even full in-memory ColBERTv2. The result serves several queries per second on a machine with a few GB of RAM, at about a quarter of the cost.

Queries Stage 1: SPLADEv2 sparse retrieval on PISA full corpus → top-200 Stage 2: ColBERT re-rank late interaction on top-200 hybrid score: SPLADE + ColBERT Results ColBERT index on disk (mmap) only touched pages enter RAM via the OS page cache
SPLADEv2 picks the candidates; ColBERT scores only those against a memory-mapped index, and the two scores are blended.

Try it. The repository includes the serving code and instructions to reproduce the memory and latency results on your own index.