# Bi-VLDoc: Bidirectional Vision-Language Modeling for Visually-Rich Document Understanding arXiv: https://arxiv.org/abs/2206.13155 ยท HTML: https://mlsanity.com/paper/2206.13155/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 43 of its 73 references that are in the map. - First posted on arXiv 27 Jun 2022. ## Placed in - Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/) ## Builds on (most cited first, 30 of 43) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017) - [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019) - [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020) - [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020) - [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019) - [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019) - [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022) - [Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression](https://mlsanity.com/paper/1911.08287/) (2019) - [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020) - [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019) - [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019) - [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021) - [UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2012.15409/) (2020) - [UNITER: UNiversal Image-TExt Representation Learning](https://mlsanity.com/paper/1909.11740/) (2019) - [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015) - [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020) - [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021) - [SelfDoc: Self-Supervised Document Representation Learning](https://mlsanity.com/paper/2106.03331/) (2021) - [Chargrid: Towards Understanding 2D Documents](https://mlsanity.com/paper/1809.08799/) (2018) - [Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network](https://mlsanity.com/paper/1706.02337/) (2017) - [StructuralLM: Structural Pre-training for Form Understanding](https://mlsanity.com/paper/2105.11210/) (2021) ## Built on by (most cited first, 18 of 18) - [LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding](https://mlsanity.com/paper/2404.05225/) (2024) - [Vision Grid Transformer for Document Layout Analysis](https://mlsanity.com/paper/2308.14978/) (2023) - [GeoLayoutLM: Geometric Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2304.10759/) (2023) - [RoDLA: Benchmarking the Robustness of Document Layout Analysis Models](https://mlsanity.com/paper/2403.14442/) (2024) - [Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understanding](https://mlsanity.com/paper/2409.19672/) (2024) - [A Simple yet Effective Layout Token in Large Language Models for Document Understanding](https://mlsanity.com/paper/2503.18434/) (2025) - [On Evaluation of Document Classification using RVL-CDIP](https://mlsanity.com/paper/2306.12550/) (2023) - [ProcTag: Process Tagging for Assessing the Efficacy of Document Instruction Data](https://mlsanity.com/paper/2407.12358/) (2024) - [DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splitting](https://mlsanity.com/paper/2602.15958/) (2026) - [Semantic Document Derendering: SVG Reconstruction via Vision-Language Modeling](https://mlsanity.com/paper/2511.13478/) (2025) - [TreeForm: End-to-end Annotation and Evaluation for Form Document Parsing](https://mlsanity.com/paper/2402.05282/) (2024) - [CrisiSense-RAG: Crisis Sensing Multimodal Retrieval-Augmented Generation for Rapid Disaster Impact Assessment](https://mlsanity.com/paper/2602.13239/) (2026) - [Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning](https://mlsanity.com/paper/2505.18603/) (2025) - [Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence](https://mlsanity.com/paper/2609.32041/) (2026) - [Enhancing Document Information Analysis with Multi-Task Pre-training: A Robust Approach for Information Extraction in Visually-Rich Documents](https://mlsanity.com/paper/2310.16527/) (2023) - [DoPTA: Improving Document Layout Analysis using Patch-Text Alignment](https://mlsanity.com/paper/2412.12902/) (2024) - [HybriDLA: Hybrid Generation for Document Layout Analysis](https://mlsanity.com/paper/2511.19919/) (2025) - [Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap](https://mlsanity.com/paper/2606.31446/) (2026) ## Abstract (arXiv) Multi-modal document pre-trained models have proven to be very effective in a variety of visually-rich document understanding (VrDU) tasks. Though existing document pre-trained models have achieved excellent performance on standard benchmarks for VrDU, the way they model and exploit the interactions between vision and language on documents has hindered them from better generalization ability and higher accuracy. In this work, we investigate the problem of vision-language joint representation learning for VrDU mainly from the perspective of supervisory signals. Specifically, a pre-training paradigm called Bi-VLDoc is proposed, in which a bidirectional vision-language supervision strategy and a vision-language hybrid-attention mechanism are devised to fully explore and utilize the interactions between these two modalities, to learn stronger cross-modal document representations with richer semantics. Benefiting from the learned informative cross-modal document representations, Bi-VLDoc significantly advances the state-of-the-art performance on three widely-used document understanding benchmarks, including Form Understanding (from 85.14% to 93.44%), Receipt Information Extraction (from 96.01% to 97.84%), and Document Classification (from 96.08% to 97.12%). On Document Visual QA, Bi-VLDoc achieves the state-of-the-art performance compared to previous single model methods.