Skip to content
anythingai.space
Retrieval & RAGadvanced12 min · 8 steps

Late Interaction & ColBERT

Keep one small vector per token and match token to token at query time — near cross-encoder accuracy with documents still encoded offline.

vectors / doc1
TYPICAL QUERY LATENCYBM25~5 msBi-encoder~10 msColBERT~30–50 msCross-encoder~500 ms+ per pairBi-encodernever interactLate interactiontoken-level MaxSimCross-encoderfull joint attention
datarequest
metrics are illustrative
1 / 8
ProblemStep 1 of 8

Fast and blurry, or sharp and slow

A bi-encoder compresses a whole document into one vector ahead of time: fast, but a rare term like “Widget-X” gets averaged away. A cross-encoder reads query and document together: most accurate, but nothing can be precomputed, so it only works for reranking a small set.

The difference is when query and document interact: never (bi-encoder), fully (cross-encoder) — or late: encode separately, then interact cheaply token by token.

Lesson 11 of 37 in Retrieval & RAG