mlsanityData to 13 Oct 2026

ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval

arXiv 2511.00903 ↗First posted 2 Nov 2025cs.CLcited by 4

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval, whether in how they encode documents, define training objectives, or compute similarity scores. To address these limitations, we present ColMate, a document retrieval model that bridges the gap between multimodal representation learning and document retrieval. ColMate utilizes a novel OCR-based pretraining objective, a self-supervised masked contrastive learning objective, and a late interaction scoring mechanism more relevant to multimodal document structures and visual characteristics. ColMate obtains 3.61% improvements over existing retrieval models on the ViDoRe V2 benchmark, demonstrating stronger generalization to out-of-domain benchmarks.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 25 of its 31 references that are in the map. First posted on arXiv 2 Nov 2025.

Builds on

25 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents14 Oct
  2. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text6 Oct

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,295
11 more
  1. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,169
  2. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 833
  3. 2020Contrastive Learning with Hard Negative Samplescited by 619
  4. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 482
  5. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  6. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 233
  7. 2022DiT: Self-supervised Pre-training for Document Image Transformercited by 152
  8. 2024Knowledge Fusion of Large Language Modelscited by 112
  9. 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 108
  10. 2022Unified Pretraining Framework for Document Understandingcited by 85
  11. 2023StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingcited by 39

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
  2. 2026MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrievalcited by 1
  3. 2026Towards Self-Explainable Document Visual Question Answering with Chain-of-Explanation Predictionscited by 1
  4. 2026ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction
  2. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval
  3. 2024ColPali: Efficient Document Retrieval with Vision Language Models
  4. 2025Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video
  5. 2026Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs
  6. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrieval
  7. 2026IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering
  8. 2026LILaC: Late Interacting in Layered Component Graph for Open-domain Multimodal Multihop Retrieval ↗