mlsanityData to 7 Oct 2026

DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming

arXiv 2406.19101 ↗First posted 27 Jun 2024cs.CVcited by 35

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Current multimodal large language models (MLLMs) face significant challenges in visual document understanding (VDU) tasks due to the high resolution, dense text, and complex layouts typical of document images. These characteristics demand a high level of detail perception ability from MLLMs. While increasing input resolution improves detail perception capability, it also leads to longer sequences of visual tokens, increasing computational costs and straining the models' ability to handle long contexts. To address these challenges, we introduce DocKylin, a document-centric MLLM that performs visual content slimming at both the pixel and token levels, thereby reducing token sequence length in VDU scenarios. We introduce an Adaptive Pixel Slimming (APS) preprocessing module to perform pixel-level slimming, increasing the proportion of informative pixels. Moreover, we propose a novel Dynamic Token Slimming (DTS) module to conduct token-level slimming, filtering essential tokens and removing others to adaptively create a more compact visual sequence. Experiments demonstrate DocKylin's promising performance across various VDU benchmarks and the effectiveness of each component.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 77 of its 91 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

77 of 91 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024HRVDA: High-Resolution Visual Document Assistant10 Apr
  2. 2024TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document7 Mar
  3. 2024Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models29 Feb
  4. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding20 Nov

Other references · 73

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
38 more
  1. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  5. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  6. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  7. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  8. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  9. 2021InfographicVQAcited by 430
  10. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  11. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  12. 2021OCR-free Document Understanding Transformercited by 400
  13. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  14. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  15. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  16. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  17. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  18. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  19. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  20. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  21. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  22. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  23. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  24. 2019Image-based table recognition: data, model, and evaluationcited by 233
  25. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  26. 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
  27. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  28. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  29. 2017Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognitioncited by 161
  30. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  31. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  32. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  33. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  34. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  35. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  36. 2017ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)cited by 103
  37. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  38. 2021StrucTexT: Structured Text Understanding with Multi-Modal Transformerscited by 90

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 16; 2026: 11.

35 papers.

Sort:

By area:

  1. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  2. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  3. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  4. 2024Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramidcited by 26
  5. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameterscited by 25
  6. 2024DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understandingcited by 23
  7. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  8. 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
  9. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  10. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  11. 2024FoPru: Focal Pruning for Efficient Large Vision-Language Modelscited by 10
  12. 2025MTabVQA: Evaluating Multi-Tabular Reasoning of Language Models in Visual Spacecited by 9
23 more
  1. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Reviewcited by 8
  2. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  3. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  4. 2025RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMscited by 6
  5. 2026DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splittingcited by 5
  6. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  7. 2026UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documentscited by 4
  8. 2024Arctic-TILT. Business Document Understanding at Sub-Billion Scalecited by 3
  9. 2026Boosting Document Parsing Efficiency and Performance with Coarse-to-Fine Visual Processingcited by 3
  10. 2025QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retrainingcited by 2
  11. 2025Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Modelscited by 1
  12. 2025MGA-VQA: Secure and Interpretable Graph-Augmented Visual Question Answering with Memory-Guided Protection Against Unauthorized Knowledge Usecited by 1
  13. 2026Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Modelscited by 1
  14. 2026CC-OCR V2: Fine-Grained Attribution of LMM Failures in Real-World Visual Document Understandingcited by 1
  15. 2026Enhancing Large Multimodal Models in Key Information Extraction via Scene-Aware Document Synthesiscited by 1
  16. 2025Document Image Rectification Bases on Self-Adaptive Multitask Fusioncited by 0
  17. 2025D2Dewarp: Dual Dimensions Geometric Representation Learning Based Document Image Dewarpingcited by 0
  18. 2025CROP: Integrating Topological and Spatial Structures via Cross-View Prefixes for Molecular LLMscited by 0
  19. 2026ROAP: A Reading-Order and Attention-Prior Pipeline for Optimizing Layout Transformers in Key Information Extractioncited by 0
  20. 2026PositionOCR: Augmenting Positional Awareness in Multi-Modal Models via Hybrid Specialist Integrationcited by 0
  21. 2026BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehensioncited by 0
  22. 2026Heterogeneous Element-Aware Cross-Version Differencing of Scientific Documents via Layout-Aware Alignment and Structure-Aware Reasoningcited by 0
  23. 2026MoTE: Mixture of Task Experts for Multi-Task Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  2. 2024AdaptVision: Dynamic Input Scaling in MLLMs for Versatile Scene Understanding
  3. 2025Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning
  4. 2025DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
  5. 2024LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models
  6. 2025Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
  7. 2026MonkeyOCRv2: A Visual-Text Foundation Model for Document AI
  8. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities