# SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion

arXiv: https://arxiv.org/abs/2503.11576 · HTML: https://mlsanity.com/paper/2503.11576/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 70 of its 102 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 30 of 70)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://mlsanity.com/paper/2309.15112/) (2023)
- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)

## Built on by (most cited first, 30 of 53)

- [SmolVLM: Redefining small and efficient multimodal models](https://mlsanity.com/paper/2504.05299/) (2025)
- [DeepSeek-OCR: Contexts Optical Compression](https://mlsanity.com/paper/2510.18234/) (2025)
- [PaddleOCR 3.0 Technical Report](https://mlsanity.com/paper/2507.05595/) (2025)
- [MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing](https://mlsanity.com/paper/2509.22186/) (2025)
- [PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model](https://mlsanity.com/paper/2510.14528/) (2025)
- [MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm](https://mlsanity.com/paper/2506.05218/) (2025)
- [Dolphin: Document Image Parsing via Heterogeneous Anchor Prompting](https://mlsanity.com/paper/2505.14059/) (2025)
- [FineVision: Open Data Is All You Need](https://mlsanity.com/paper/2510.17269/) (2025)
- [dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model](https://mlsanity.com/paper/2512.02498/) (2025)
- [POINTS-Reader: Distillation-Free Adaptation of Vision-Language Models for Document Conversion](https://mlsanity.com/paper/2509.01215/) (2025)
- [Dolphin-v2: Universal Document Parsing via Scalable Anchor Prompting](https://mlsanity.com/paper/2602.05384/) (2026)
- [Logics-Parsing Technical Report](https://mlsanity.com/paper/2509.19760/) (2025)
- [UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters](https://mlsanity.com/paper/2512.21095/) (2025)
- [Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding](https://mlsanity.com/paper/2510.15253/) (2025)
- [DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation](https://mlsanity.com/paper/2511.18434/) (2025)
- [ChartGen: Scaling Chart Understanding Via Code-Guided Synthetic Chart Generation](https://mlsanity.com/paper/2507.19492/) (2025)
- [Advanced Layout Analysis Models for Docling](https://mlsanity.com/paper/2509.11720/) (2025)
- [ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision](https://mlsanity.com/paper/2602.14276/) (2026)
- [MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures](https://mlsanity.com/paper/2503.16096/) (2025)
- [DvD: Unleashing a Generative Paradigm for Document Dewarping via Coordinates-based Diffusion Model](https://mlsanity.com/paper/2505.21975/) (2025)
- [Baseer: A Vision-Language Model for Arabic Document-to-Markdown OCR](https://mlsanity.com/paper/2509.18174/) (2025)
- [Uni-Parser Technical Report](https://mlsanity.com/paper/2512.15098/) (2025)
- [Towards Reliable and Interpretable Document Question Answering via VLMs](https://mlsanity.com/paper/2509.10129/) (2025)
- [PubTables-v2: A new large-scale dataset for full-page and multi-page table extraction](https://mlsanity.com/paper/2512.10888/) (2025)
- [DAVE: A VLM Vision Encoder for Document Understanding and Web Agents](https://mlsanity.com/paper/2512.17221/) (2025)
- [GutenOCR: A Grounded Vision-Language Front-End for Documents](https://mlsanity.com/paper/2601.14490/) (2026)
- [MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures](https://mlsanity.com/paper/2603.28550/) (2026)
- [E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognition](https://mlsanity.com/paper/2509.03615/) (2025)
- [Multi-Modal Vision vs. Text-Based Parsing: Benchmarking LLM Strategies for Invoice Processing](https://mlsanity.com/paper/2509.04469/) (2025)
- [NVIDIA Nemotron Parse 1.1](https://mlsanity.com/paper/2511.20478/) (2025)

## Abstract (arXiv)

We introduce SmolDocling, an ultra-compact vision-language model targeting end-to-end document conversion. Our model comprehensively processes entire pages by generating DocTags, a new universal markup format that captures all page elements in their full context with location. Unlike existing approaches that rely on large foundational models, or ensemble solutions that rely on handcrafted pipelines of multiple specialized models, SmolDocling offers an end-to-end conversion for accurately capturing content, structure and spatial location of document elements in a 256M parameters vision-language model. SmolDocling exhibits robust performance in correctly reproducing document features such as code listings, tables, equations, charts, lists, and more across a diverse range of document types including business documents, academic papers, technical reports, patents, and forms -- significantly extending beyond the commonly observed focus on scientific papers. Additionally, we contribute novel publicly sourced datasets for charts, tables, equations, and code recognition. Experimental results demonstrate that SmolDocling competes with other Vision Language Models that are up to 27 times larger in size, while reducing computational requirements substantially. The model is currently available, datasets will be publicly available soon.