mlsanityData to 7 Oct 2026

SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion

arXiv 2503.11576 ↗First posted 14 Mar 2025cs.CVcited by 51

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

We introduce SmolDocling, an ultra-compact vision-language model targeting end-to-end document conversion. Our model comprehensively processes entire pages by generating DocTags, a new universal markup format that captures all page elements in their full context with location. Unlike existing approaches that rely on large foundational models, or ensemble solutions that rely on handcrafted pipelines of multiple specialized models, SmolDocling offers an end-to-end conversion for accurately capturing content, structure and spatial location of document elements in a 256M parameters vision-language model. SmolDocling exhibits robust performance in correctly reproducing document features such as code listings, tables, equations, charts, lists, and more across a diverse range of document types including business documents, academic papers, technical reports, patents, and forms -- significantly extending beyond the commonly observed focus on scientific papers. Additionally, we contribute novel publicly sourced datasets for charts, tables, equations, and code recognition. Experimental results demonstrate that SmolDocling competes with other Vision Language Models that are up to 27 times larger in size, while reducing computational requirements substantially. The model is currently available, datasets will be publicly available soon.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 70 of its 102 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

70 of 102 references.

References · 70

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024Mixtral of Expertscited by 1,785
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  8. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  9. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  10. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  11. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  12. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  13. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  14. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  15. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  16. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  17. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  18. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  19. 2023Nougat: Neural Optical Understanding for Academic Documentscited by 240
  20. 2019Image-based table recognition: data, model, and evaluationcited by 233
  21. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  22. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  23. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  24. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  25. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  26. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  27. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  28. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  29. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  30. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  31. 2020Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Contextcited by 146
  32. 2022DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysiscited by 146
  33. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  34. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  35. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  36. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  37. 2021PubTables-1M: Towards comprehensive table extraction from unstructured documentscited by 122
  38. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 30; 2026: 21.

51 papers.

Sort:

By area:

  1. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  2. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 174
  3. 2025PaddleOCR 3.0 Technical Reportcited by 136
  4. 2025MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsingcited by 108
  5. 2025PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Modelcited by 84
  6. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 81
  7. 2025Dolphin: Document Image Parsing via Heterogeneous Anchor Promptingcited by 78
  8. 2025FineVision: Open Data Is All You Needcited by 51
  9. 2025dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Modelcited by 44
  10. 2025POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversioncited by 34
  11. 2026Dolphin-v2: Universal Document Parsing via Scalable Anchor Promptingcited by 20
  12. 2025Logics-Parsing Technical Reportcited by 16
38 more
  1. 2025UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameterscited by 12
  2. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  3. 2025DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translationcited by 8
  4. 2025ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generationcited by 7
  5. 2025Advanced Layout Analysis Models for Doclingcited by 6
  6. 2025MarkushGrapher: Joint Visual and Textual Recognition of Markush Structurescited by 5
  7. 2025DvD: Unleashing a Generative Paradigm for Document Dewarping via Coordinates-based Diffusion Modelcited by 4
  8. 2025Baseer: A Vision-Language Model for Arabic Document-to-Markdown OCRcited by 4
  9. 2026ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervisioncited by 4
  10. 2025Uni-Parser Technical Reportcited by 3
  11. 2025Towards Reliable and Interpretable Document Question Answering via VLMscited by 2
  12. 2025PubTables-v2: A new large-scale dataset for full-page and multi-page table extractioncited by 2
  13. 2025DAVE: A VLM Vision Encoder for Document Understanding and Web Agentscited by 2
  14. 2026GutenOCR: A Grounded Vision-Language Front-End for Documentscited by 2
  15. 2026MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structurescited by 2
  16. 2025E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognitioncited by 1
  17. 2025Multi-Modal Vision vs. Text-Based Parsing: Benchmarking LLM Strategies for Invoice Processingcited by 1
  18. 2025NVIDIA Nemotron Parse 1.1cited by 1
  19. 2025CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusioncited by 1
  20. 2026TDATR: Improving End-to-End Table Recognition via Table Detail-Aware Learning and Cell-Level Visual Alignmentcited by 1
  21. 2026Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Modelscited by 1
  22. 2026TexOCR: Advancing Document OCR Models for Compilable Page-to-LaTeX Reconstructioncited by 1
  23. 2026TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?cited by 1
  24. 2026PaDoc: Layout-Grounded Parallel Decoding for Document Parsingcited by 1
  25. 2025SubGrapher: Visual Fingerprinting of Chemical Structurescited by 0
  26. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0
  27. 2026VERSE: Visual Embedding Reduction and Space Exploration. Clustering-Guided Insights for Training Data Enhancement in Visually-Rich Document Understandingcited by 0
  28. 2025Clinical Document Metadata Extraction: A Scoping Reviewcited by 0
  29. 2026DocAtlas: Multilingual Document Understanding Across 80+ Languagescited by 0
  30. 2026Structured Layout Priors for Robust Out-of-Distribution Visual Document Understandingcited by 0
  31. 2026POTATR: A Lightweight Image-to-Graph Model for Page-Level Table Extractioncited by 0
  32. 2026Active Learning for Cascaded Object Detection: Balancing Coverage and Uncertainty in Table Extraction Pipelinescited by 0
  33. 2026Identify, Locate, Link: End-to-End Key-Value Extraction from Document Imagescited by 0
  34. 2026Jina-OCR-v1: Efficient Document Parsing with Speculative Decoding and Dense Verifiable Rewardscited by 0
  35. 2026Tables Decoded: DELTA for Structure, TARQA for Understandingcited by 0
  36. 2026WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsingcited by 0
  37. 2026DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answeringcited by 0
  38. 2026Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QAcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Vision-Language Models for French PDF-to-Markdown Conversion
  2. 2026Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild
  3. 2026SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
  4. 2026Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training
  5. 2026MonkeyOCRv2: A Visual-Text Foundation Model for Document AI
  6. 2025Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding
  7. 2025MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding
  8. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding