mlsanityData to 7 Oct 2026

ColPali: Efficient Document Retrieval with Vision Language Models

arXiv 2407.01449 ↗First posted 27 Jun 2024cs.IR cs.CL cs.CVcited by 190

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Documents are visually rich structures that convey information through text, but also figures, page layouts, tables, or even fonts. Since modern retrieval systems mainly rely on the textual information they extract from document pages to index documents -often through lengthy and brittle processes-, they struggle to exploit key visual cues efficiently. This limits their capabilities in many practical document retrieval applications such as Retrieval Augmented Generation (RAG). To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings. The inherent complexity and performance shortcomings of modern systems motivate a new concept; doing document retrieval by directly embedding the images of the document pages. We release ColPali, a Vision Language Model trained to produce high-quality multi-vector embeddings from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically simpler, faster and end-to-end trainable. We release models, data, code and benchmarks under open licenses at https://hf.co/vidore.

Key facts. Cited by 190 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 75 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

42 of 75 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
30 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  7. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  8. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  9. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,064
  10. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  11. 2023Vision Transformers Need Registerscited by 844
  12. 2023Large Language Modelscited by 774
  13. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  14. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  15. 2021OCR-free Document Understanding Transformercited by 400
  16. 2024What matters when building vision-language models?cited by 321
  17. 2023Improving Text Embeddings with Large Language Modelscited by 295
  18. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  19. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  20. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  21. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  22. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  23. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  24. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  25. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  26. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  27. 2024Jina CLIP: Your CLIP Model Is Also Your Text Retrievercited by 46
  28. 2019Table2Vec: Neural Word and Entity Embeddings for Table Population and Retrievalcited by 37
  29. 2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 28
  30. 2024Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanismcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 68; 2026: 115.

190 papers.

Sort:

By area:

  1. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  2. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  3. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  4. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  5. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  6. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
  7. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 76
  8. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  9. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 69
  10. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  11. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 66
  12. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
38 more
  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  2. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  3. 2025UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddingscited by 37
  4. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  5. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
  6. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  7. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  8. 2024VISA: Retrieval Augmented Generation with Visual Source Attributioncited by 28
  9. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 28
  10. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  11. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 24
  12. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 22
  13. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  14. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  15. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  16. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  17. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 19
  18. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  19. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 18
  20. 2025FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domaincited by 16
  21. 2025MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphscited by 16
  22. 2025DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understandingcited by 15
  23. 2024PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Samplingcited by 14
  24. 2025Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learningcited by 13
  25. 2026e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddingscited by 12
  26. 2025Closing the Modality Gap for Mixed Modality Searchcited by 11
  27. 2025Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimizationcited by 10
  28. 2025CLaMR: Contextualized Late-Interaction for Multimodal Content Retrievalcited by 9
  29. 2025VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generationcited by 9
  30. 2025IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrievalcited by 8
  31. 2025One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Imagecited by 8
  32. 2025Fill the Gap: Quantifying and Reducing the Modality Gap in Image-Text Representation Learningcited by 8
  33. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  34. 2025RegionRAG: Region-level Retrieval-Augmented Generation for Visual Document Understandingcited by 8
  35. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  36. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Frameworkcited by 7
  37. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
  38. 2025Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Rerankingcited by 6

50 most cited of 190 All 190 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DocPC: Document-Level Visual Retrieval via Representative Page Composition
  2. 2025Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction
  3. 2025SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models
  4. 2026NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval
  5. 2025MIRACL-VISION: A Large, multilingual, visual document retrieval benchmark
  6. 2026Nemotron ColEmbed V2: Top-Performing Late Interaction Embedding Models for Visual Document Retrieval
  7. 2026Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval
  8. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings