DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiency
Not on the map. Why?
Abstract (arXiv)
This paper introduces DONUT-hole, a sparse OCR-free visual document understanding (VDU) model that addresses the limitations of its predecessor model, dubbed DONUT. The DONUT model, leveraging a transformer architecture, overcoming the challenges of separate optical character recognition (OCR) and visual semantic understanding (VSU) components. However, its deployment in production environments and edge devices is hindered by high memory and computational demands, particularly in large-scale request services. To overcome these challenges, we propose an optimization strategy based on knowledge distillation and model pruning. Our paradigm to produce DONUT-hole, reduces the model denisty by 54% while preserving performance. We also achieve a global representational similarity index between DONUT and DONUT-hole based on centered kernel alignment (CKA) metric of 0.79. Moreover, we evaluate the effectiveness of DONUT-hole in the document image key information extraction (KIE) task, highlighting its potential for developing more efficient VDU systems for logistic companies.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 23 references that are in the map. First posted on arXiv 9 Nov 2023.
Builds on
20 of 23 references.
References · 20
- 2017Attention Is All You Needcited by 60,361
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2016Sequence-Level Knowledge Distillationcited by 801
- 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
- 2013Do Deep Nets Really Need to be Deep?cited by 474
- 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
- 2021OCR-free Document Understanding Transformercited by 407
- 2021Distilling Knowledge via Knowledge Reviewcited by 288
- 2022Knowledge Distillation from A Stronger Teachercited by 208
- 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
8 more
- 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
- 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
- 2019Graph Convolution for Multimodal Information Extraction from Visually Rich Documentscited by 88
- 2020PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networkscited by 72
- 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
- 2021Spatial Dual-Modality Graph Reasoning for Key Information Extractioncited by 43
- 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
- 2022TransKD: Transformer Knowledge Distillation for Efficient Semantic Segmentationcited by 17
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understanding
- 2025A Token-level Text Image Foundation Model for Document Understanding
- 2026Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search
- 2024Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding
- 2025Index-Preserving Lightweight Token Pruning for Efficient Document Understanding in Vision-Language Models
- 2022Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding
- 2025VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization
- 2026Visual Information Extraction from Documents via Classification-Guided Large Vision-Language Models