mlsanityData to 14 Oct 2026

Interpret, prune and distill Donut : towards lightweight VLMs for VQA on document

arXiv 2509.26235 ↗First posted 30 Sep 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Recent advances in Visually-rich Document Understanding rely on large Vision-Language Models like Donut, which perform document-level Visual Question Answering without Optical Character Recognition. Despite their effectiveness, these models are too costly for real-time or resource-constrained applications. We investigate model compression through knowledge distillation, training compact student models from a larger teacher. We leverage mechanistic interpretability to drive student architecture design within this framework. By analyzing internal computations, we identify essential subcomponents to retain, while having a clear view of which subcomponents should be approximated, skipped, or reparametrized based on their function. This approach yields Donut-MINT (Mechanistic Interpretability-based Network Trimming), a pruned Donut variant that reduces inference time and memory usage while maintaining strong performance on DocVQA, a standard benchmark for document Visual Question Answering. Our method reframes compression as circuit discovery, bridging interpretability research and practical Vision-Language Model deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 56 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

47 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models30 Nov
  2. 2023SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models24 May

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  9. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  11. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
33 more
  1. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  3. 2022In-context Learning and Induction Headscited by 964
  4. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  5. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  6. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  7. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  8. 2019Similarity-Preserving Knowledge Distillationcited by 516
  9. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  10. 2021OCR-free Document Understanding Transformercited by 407
  11. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  13. 2019Scene Text Visual Question Answeringcited by 331
  14. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 279
  15. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  16. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  17. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  18. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  19. 2022PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR Systemcited by 106
  20. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  21. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  22. 2023Localizing and Editing Knowledge in Text-to-Image Generative Modelscited by 54
  23. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Modelscited by 49
  24. 2024SlimGPT: Layer-wise Structured Pruning for Large Language Modelscited by 44
  25. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  26. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Modelscited by 27
  27. 2023Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanismcited by 22
  28. 2024An Investigation of Neuron Activation as a Unified Lens to Explain Chain-of-Thought Eliciting Arithmetic Reasoning of LLMscited by 20
  29. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answeringcited by 11
  30. 2024Interpreting Neurons in Deep Vision Networks with Language Modelscited by 10
  31. 2025Beyond Intermediate States: Explaining Visual Redundancy through Languagecited by 7
  32. 2025FASP: Fast and Accurate Structured Pruning of Large Language Modelscited by 6
  33. 2023DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiencycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024GRAM: Global Reasoning for Multi-Page VQA
  2. 2023Dynamic Clue Bottlenecks: Towards Interpretable-by-Design Visual Question Answering
  3. 2026Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering
  4. 2024Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering
  5. 2025QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQA
  6. 2026TestHallVQA: Exploring LVLMs' Document-Level Reasoning under Redundant Contexts from Scientific Exams
  7. 2024Efficient Large Multi-modal Models via Visual Context Compression
  8. 2026Rethinking Token Reduction for Large Vision-Language Models