# Multimodal deep networks for text and image-based document classification

arXiv: https://arxiv.org/abs/1907.06370 · HTML: https://mlsanity.com/paper/1907.06370/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2019.

## Built on by (most cited first, 24 of 24)

- [Benchmarking Multimodal AutoML for Tabular Data with Text Fields](https://mlsanity.com/paper/2111.02705/) (2021)
- [Multimodal Pre-training Based on Graph Attention Network for Document Understanding](https://mlsanity.com/paper/2203.13530/) (2022)
- [Form2Seq : A Framework for Higher-Order Form Structure Extraction](https://mlsanity.com/paper/2107.04419/) (2021)
- [Evaluating Out-of-Distribution Performance on Document Image Classifiers](https://mlsanity.com/paper/2210.07448/) (2022)
- [Name Your Style: An Arbitrary Artist-aware Image Style Transfer](https://mlsanity.com/paper/2202.13562/) (2022)
- [VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classification](https://mlsanity.com/paper/2205.12029/) (2022)
- [Deep Learning for Technical Document Classification](https://mlsanity.com/paper/2106.14269/) (2021)
- [Multi-Modal Association based Grouping for Form Structure Extraction](https://mlsanity.com/paper/2107.04396/) (2021)
- [On Evaluation of Document Classification using RVL-CDIP](https://mlsanity.com/paper/2306.12550/) (2023)
- [Semantic Parsing of Interpage Relations](https://mlsanity.com/paper/2205.13530/) (2022)
- [EAML: Ensemble Self-Attention-based Mutual Learning Network for Document Image Classification](https://mlsanity.com/paper/2305.06923/) (2023)
- [Efficient Document Image Classification Using Region-Based Graph Neural Network](https://mlsanity.com/paper/2106.13802/) (2021)
- [Sequence-aware multimodal page classification of Brazilian legal documents](https://mlsanity.com/paper/2207.00748/) (2022)
- [On-Device Document Classification using multimodal features](https://mlsanity.com/paper/2101.01880/) (2021)
- [Unimodal and Multimodal Representation Training for Relation Extraction](https://mlsanity.com/paper/2211.06168/) (2022)
- [Multimodal Side-Tuning for Document Classification](https://mlsanity.com/paper/2301.07502/) (2023)
- [STORM: Strategic Orchestration of Modalities for Rare Event Classification](https://mlsanity.com/paper/2412.02805/) (2024)
- [Deep learning using Havrda-Charvat entropy for classification of pulmonary endomicroscopy](https://mlsanity.com/paper/2104.05450/) (2021)
- [End-to-End Document Classification and Key Information Extraction using Assignment Optimization](https://mlsanity.com/paper/2306.00750/) (2023)
- [OPAD: An Optimized Policy-based Active Learning Framework for Document Content Analysis](https://mlsanity.com/paper/2110.02069/) (2021)
- [Bridging Text and Image for Artist Style Transfer via Contrastive Learning](https://mlsanity.com/paper/2410.09566/) (2024)
- [WordVIS: A Color Worth A Thousand Words](https://mlsanity.com/paper/2412.10155/) (2024)
- [DEMUN: Fast and accurate discovery of music notation in very large collections](https://mlsanity.com/paper/2606.31956/) (2026)
- [DocHRL: A Hierarchical Reinforcement Learning Framework for Cost-Optimised Document Classification](https://mlsanity.com/paper/2607.22644/) (2026)

## Abstract (arXiv)

Classification of document images is a critical step for archival of old manuscripts, online subscription and administrative procedures. Computer vision and deep learning have been suggested as a first solution to classify documents based on their visual appearance. However, achieving the fine-grained classification that is required in real-world setting cannot be achieved by visual analysis alone. Often, the relevant information is in the actual text content of the document. We design a multimodal neural network that is able to learn from word embeddings, computed on text extracted by OCR, and from the image. We show that this approach boosts pure image accuracy by 3% on Tobacco3482 and RVL-CDIP augmented by our new QS-OCR text dataset (https://github.com/Quicksign/ocrized-text-dataset), even without clean text information.