mlsanityData to 14 Oct 2026

VERSE: Visual Embedding Reduction and Space Exploration. Clustering-Guided Insights for Training Data Enhancement in Visually-Rich Document Understanding

arXiv 2601.05125 ↗First posted 8 Jan 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

This work introduces VERSE, a methodology for analyzing and improving Vision-Language Models applied to Visually-rich Document Understanding by exploring their visual embedding space. VERSE enables the visualization of latent representations, supporting the assessment of model feasibility. It also facilitates the identification of problematic regions and guides the generation of synthetic data to enhance performance in those clusters. We validate the methodology by training on the synthetic MERIT Dataset and evaluating on its real-world counterpart, MERIT Secret. Results show that VERSE helps uncover the visual features associated with error-prone clusters, and that retraining with samples containing these features substantially boosts F1 performance without degrading generalization. Furthermore, we demonstrate that on-premise models such as Donut and Idefics2, when optimized with VERSE, match or even surpass the performance of SaaS solutions like GPT-4 and Pixtral.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 44 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

34 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion14 Mar

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  7. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  8. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  9. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  10. 2021InfographicVQAcited by 437
  11. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  12. 2021OCR-free Document Understanding Transformercited by 407
21 more
  1. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  2. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
  3. 2024What matters when building vision-language models?cited by 326
  4. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  5. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  6. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 254
  7. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 179
  8. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  9. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  10. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  11. 2022XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understandingcited by 68
  12. 2022All You May Need for VQA are Image Captionscited by 65
  13. 2025Docling: An Efficient Open-Source Toolkit for AI-driven Document Conversioncited by 63
  14. 2022OCR-IDL: OCR Annotations for Industry Document Library Datasetcited by 35
  15. 2023Optimized Table Tokenization for Table Structure Recognitioncited by 34
  16. 2023PDFVQA: A New Dataset for Real-World VQA on PDF Documentscited by 28
  17. 2019NNE: A Dataset for Nested Named Entity Recognition in English Newswirecited by 25
  18. 2023DocParser: End-to-end OCR-free Information Extraction from Visually Rich Documentscited by 19
  19. 2024CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentationcited by 14
  20. 2024Transformers Use Causal World Models in Maze-Solving Taskscited by 13
  21. 2024The MERIT Dataset: Modelling and Efficiently Rendering Interpretable Transcriptscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
  2. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  3. 2026Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval
  4. 2023Learning the Visualness of Text Using Large Vision-Language Models
  5. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming
  6. 2025QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understanding
  7. 2026MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval
  8. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning