# A Study on Token Pruning for ColBERT

arXiv: https://arxiv.org/abs/2112.06540 · HTML: https://mlsanity.com/paper/2112.06540/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving.
- Builds on 16 of its 20 references that are in the map.
- First posted on arXiv 13 Dec 2021.

## Placed in

- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://mlsanity.com/paper/2108.05540/) (2021)
- [SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval](https://mlsanity.com/paper/2109.10086/) (2021)
- [Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search](https://mlsanity.com/paper/2010.07003/) (2020)
- [Efficient Passage Retrieval with Hashing for Open-domain Question Answering](https://mlsanity.com/paper/2106.00882/) (2021)
- [Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval](https://mlsanity.com/paper/2106.11251/) (2021)
- [Mitigating the Position Bias of Transformer Models in Passage Re-Ranking](https://mlsanity.com/paper/2101.06980/) (2021)
- [Query Embedding Pruning for Dense Retrieval](https://mlsanity.com/paper/2108.10341/) (2021)

## Built on by (most cited first, 15 of 15)

- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Token Fusion: Bridging the Gap between Token Pruning and Token Merging](https://mlsanity.com/paper/2312.01026/) (2023)
- [CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers](https://mlsanity.com/paper/2305.17455/) (2023)
- [Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding](https://mlsanity.com/paper/2407.14439/) (2024)
- [Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction](https://mlsanity.com/paper/2203.13088/) (2022)
- [iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models](https://mlsanity.com/paper/2412.06263/) (2024)
- [Multi-Vector Index Compression in Any Modality](https://mlsanity.com/paper/2602.21202/) (2026)
- [PaPr: Training-Free One-Step Patch Pruning with Lightweight ConvNets for Faster Inference](https://mlsanity.com/paper/2403.16020/) (2024)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [LightVLM: Acceleraing Large Multimodal Models with Pyramid Token Merging and KV Cache Compression](https://mlsanity.com/paper/2509.00419/) (2025)
- [EigenLI: Spectral Approximations to Late Interaction](https://mlsanity.com/paper/2609.07561/) (2026)
- [ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition](https://mlsanity.com/paper/2412.16491/) (2024)
- [Query-Aware Token Budgeting for Efficient Late-Interaction Visual Document Retrieval](https://mlsanity.com/paper/2609.07262/) (2026)
- [Region-Aware CLS Token Augmentation for Fine-Grained Image Retrieval](https://mlsanity.com/paper/2610.10991/) (2026)
- [Simple Linear Corrections for Compression in Late Interaction](https://mlsanity.com/paper/2610.14196/) (2026)

## Abstract (arXiv)

The ColBERT model has recently been proposed as an effective BERT based ranker. By adopting a late interaction mechanism, a major advantage of ColBERT is that document representations can be precomputed in advance. However, the big downside of the model is the index size, which scales linearly with the number of tokens in the collection. In this paper, we study various designs for ColBERT models in order to attack this problem. While compression techniques have been explored to reduce the index size, in this paper we study token pruning techniques for ColBERT. We compare simple heuristics, as well as a single layer of attention mechanism to select the tokens to keep at indexing time. Our experiments show that ColBERT indexes can be pruned up to 30% on the MS MARCO passage collection without a significant drop in performance. Finally, we experiment on MS MARCO documents, which reveal several challenges for such mechanism.