# Image and Encoded Text Fusion for Multi-Modal Classification

arXiv: https://arxiv.org/abs/1810.02001 · HTML: https://mlsanity.com/paper/1810.02001/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2018.

## Built on by (most cited first, 9 of 9)

- [Single-branch Network for Multimodal Training](https://mlsanity.com/paper/2303.06129/) (2023)
- [EAML: Ensemble Self-Attention-based Mutual Learning Network for Document Image Classification](https://mlsanity.com/paper/2305.06923/) (2023)
- [Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach](https://mlsanity.com/paper/2408.07445/) (2024)
- [A Comparative Study of Speaker Role Identification in Air Traffic Communication Using Deep Learning Approaches](https://mlsanity.com/paper/2111.02041/) (2021)
- [Detection of Propaganda Techniques in Visuo-Lingual Metaphor in Memes](https://mlsanity.com/paper/2205.02937/) (2022)
- [Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities](https://mlsanity.com/paper/2407.16243/) (2024)
- [MATCHED: Multimodal Authorship-Attribution To Combat Human Trafficking in Escort-Advertisement Data](https://mlsanity.com/paper/2412.13794/) (2024)
- [Beyond Images: Adaptive Fusion of Visual and Textual Data for Food Classification](https://mlsanity.com/paper/2308.02562/) (2023)
- [TrendGen: An Outfit Recommendation and Display System](https://mlsanity.com/paper/2603.27264/) (2026)

## Abstract (arXiv)

Multi-modal approaches employ data from multiple input streams such as textual and visual domains. Deep neural networks have been successfully employed for these approaches. In this paper, we present a novel multi-modal approach that fuses images and text descriptions to improve multi-modal classification performance in real-world scenarios. The proposed approach embeds an encoded text onto an image to obtain an information-enriched image. To learn feature representations of resulting images, standard Convolutional Neural Networks (CNNs) are employed for the classification task. We demonstrate how a CNN based pipeline can be used to learn representations of the novel fusion approach. We compare our approach with individual sources on two large-scale multi-modal classification datasets while obtaining encouraging results. Furthermore, we evaluate our approach against two famous multi-modal strategies namely early fusion and late fusion.