# Multimodal Side-Tuning for Document Classification

arXiv: https://arxiv.org/abs/2301.07502 · HTML: https://mlsanity.com/paper/2301.07502/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 16 Jan 2023.

## Builds on (most cited first, 21 of 21)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [Multimodal Deep Learning for Robust RGB-D Object Recognition](https://mlsanity.com/paper/1507.06821/) (2015)
- [Multimodal deep networks for text and image-based document classification](https://mlsanity.com/paper/1907.06370/) (2019)
- [Cutting the Error by Half: Investigation of Very Deep CNN and Advanced Training Strategies for Document Image Classification](https://mlsanity.com/paper/1704.03557/) (2017)
- [Document Image Classification with Intra-Domain Transfer Learning and Stacked Generalization of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1801.09321/) (2018)
- [Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks](https://mlsanity.com/paper/1912.13503/) (2019)
- [Real-Time Document Image Classification using Deep CNN and Extreme Learning Machines](https://mlsanity.com/paper/1711.05862/) (2017)
- [What is the right way to represent document images?](https://mlsanity.com/paper/1603.01076/) (2016)
- [Page Stream Segmentation with Convolutional Neural Nets Combining Textual and Visual Features](https://mlsanity.com/paper/1710.03006/) (2017)

## Built on by (most cited first, 2 of 2)

- [On Evaluation of Document Classification using RVL-CDIP](https://mlsanity.com/paper/2306.12550/) (2023)
- [Document Classification Pattern Recognition via Information Fusion: A Systematic Review of Multimodal and Multiview Representation Approaches](https://mlsanity.com/paper/2605.23910/) (2026)

## Abstract (arXiv)

In this paper, we propose to exploit the side-tuning framework for multimodal document classification. Side-tuning is a methodology for network adaptation recently introduced to solve some of the problems related to previous approaches. Thanks to this technique it is actually possible to overcome model rigidity and catastrophic forgetting of transfer learning by fine-tuning. The proposed solution uses off-the-shelf deep learning architectures leveraging the side-tuning framework to combine a base model with a tandem of two side networks. We show that side-tuning can be successfully employed also when different data sources are considered, e.g. text and images in document classification. The experimental results show that this approach pushes further the limit for document classification accuracy with respect to the state of the art.