mlsanityData to 7 Oct 2026

Unifying Multimodal Retrieval via Document Screenshot Embedding

arXiv 2406.11251 ↗First posted 17 Jun 2024cs.IRcited by 107

Not on the map. Why?

Cited by 31 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the real world, documents are organized in different formats and varied modalities. Traditional retrieval pipelines require tailored document parsing techniques and content extraction modules to prepare input for indexing. This process is tedious, prone to errors, and has information loss. To this end, we propose Document Screenshot Embedding (DSE), a novel retrieval paradigm that regards document screenshots as a unified input format, which does not require any content extraction preprocess and preserves all the information in a document (e.g., text, image and layout). DSE leverages a large vision-language model to directly encode document screenshots into dense representations for retrieval. To evaluate our method, we first craft the dataset of Wiki-SS, a 1.3M Wikipedia web page screenshots as the corpus to answer the questions from the Natural Questions dataset. In such a text-intensive document retrieval setting, DSE shows competitive effectiveness compared to other text retrieval methods relying on parsing. For example, DSE outperforms BM25 by 17 points in top-1 retrieval accuracy. Additionally, in a mixed-modality task of slide retrieval, DSE significantly outperforms OCR text retrieval methods by over 15 points in nDCG@10. These experiments show that DSE is an effective document retrieval paradigm for diverse types of documents. Model checkpoints, code, and Wiki-SS collection will be released.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jun 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 48; 2026: 53.

107 papers.

Sort:

By area:

  1. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  2. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  3. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  4. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  5. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 98
  6. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  7. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  8. 2025MMDocIR: Benchmarking Multimodal Retrieval for Long Documentscited by 43
  9. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  10. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  11. 2025REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmarkcited by 34
  12. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 33
38 more
  1. 2025A Survey of Data Agents: Emerging Paradigm or Overstated Hype?cited by 30
  2. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  3. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  4. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  5. 2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrievalcited by 21
  6. 2025MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddingscited by 20
  7. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  8. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  9. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 19
  10. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAGcited by 18
  11. 2025FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domaincited by 16
  12. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  13. 2026e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddingscited by 12
  14. 2025Memory-QA: Answering Recall Questions Based on Multimodal Memoriescited by 10
  15. 2025MIRACL-VISION: A Large, multilingual, visual document retrieval benchmarkcited by 9
  16. 2025DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankerscited by 9
  17. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  18. 2025Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrievalcited by 8
  19. 2026Nemotron ColEmbed V2: Top-Performing Late Interaction Embedding Models for Visual Document Retrievalcited by 8
  20. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 8
  21. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  22. 2025Generalized Contrastive Learning for Universal Multimodal Retrievalcited by 7
  23. 2024Unified Multimodal Interleaved Document Representation for Retrievalcited by 6
  24. 2025Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Rerankingcited by 6
  25. 2025MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generationcited by 6
  26. 2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embeddingcited by 6
  27. 2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrievalcited by 6
  28. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  29. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  30. 2025URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understandingcited by 5
  31. 2025HKRAG: Holistic Knowledge Retrieval-Augmented Generation Over Visually-Rich Documentscited by 5
  32. 2025Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmarkcited by 5
  33. 2025Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracycited by 4
  34. 2025ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrievalcited by 4
  35. 2025SDS KoPub VDR: A Benchmark Dataset for Visual Document Retrieval in Korean Public Documentscited by 4
  36. 2025PixelWorld: How Far Are We from Perceiving Everything as Pixels?cited by 3
  37. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Modelscited by 3
  38. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval
  2. 2026IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering
  3. 2023AToMiC: An Image/Text Retrieval Test Collection to Support Multimedia Content Creation
  4. 2024ColPali: Efficient Document Retrieval with Vision Language Models
  5. 2025SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models
  6. 2025ABC: Achieving Better Control of Multimodal Embeddings using VLMs
  7. 2025Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video
  8. 2025DocMMIR: A Framework for Document Multi-modal Information Retrieval