# Multimodal Classification: Current Landscape, Taxonomy and Future Directions

arXiv: https://arxiv.org/abs/2109.09020 · HTML: https://mlsanity.com/paper/2109.09020/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 194 references that are in the map.
- First posted on arXiv 18 Sep 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Cost Sensitive Learning of Deep Feature Representations from Imbalanced Data](https://mlsanity.com/paper/1508.03422/) (2015)
- [A Survey on Multi-view Learning](https://mlsanity.com/paper/1304.5634/) (2013)
- [More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification](https://mlsanity.com/paper/2008.05457/) (2020)
- [Deep Learning in Medical Image Registration: A Survey](https://mlsanity.com/paper/1903.02026/) (2019)
- [A Survey of Multi-View Representation Learning](https://mlsanity.com/paper/1610.01206/) (2016)
- [Multimodal Intelligence: Representation Learning, Information Fusion, and Applications](https://mlsanity.com/paper/1911.03977/) (2019)
- [Deep Learning for Classification of Hyperspectral Data: A Comparative Review](https://mlsanity.com/paper/1904.10674/) (2019)
- [Multimodal Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/2011.08899/) (2020)
- [Multimodal Machine Learning for Automated ICD Coding](https://mlsanity.com/paper/1810.13348/) (2018)
- [A Survey on Multi-View Clustering](https://mlsanity.com/paper/1712.06246/) (2017)
- [EmbraceNet: A robust deep learning architecture for multimodal classification](https://mlsanity.com/paper/1904.09078/) (2019)
- [Temporal Multimodal Fusion for Video Emotion Classification in the Wild](https://mlsanity.com/paper/1709.07200/) (2017)
- [A review on deep learning techniques for 3D sensed data classification](https://mlsanity.com/paper/1907.04444/) (2019)
- [Combined Cleaning and Resampling Algorithm for Multi-Class Imbalanced Data with Label Noise](https://mlsanity.com/paper/2004.03406/) (2020)
- [AENet: Learning Deep Audio Features for Video Analysis](https://mlsanity.com/paper/1701.00599/) (2017)
- [Multimodal music information processing and retrieval: survey and future challenges](https://mlsanity.com/paper/1902.05347/) (2019)
- [Multimodal Emotion Classification](https://mlsanity.com/paper/1903.12520/) (2019)
- [ECG Heartbeat Classification Using Multimodal Fusion](https://mlsanity.com/paper/2107.09869/) (2021)
- [Multimodal deep learning approach for joint EEG-EMG data compression and classification](https://mlsanity.com/paper/1703.08970/) (2017)
- [A Joint Model for Multimodal Document Quality Assessment](https://mlsanity.com/paper/1901.01010/) (2019)
- [Multimodal Age and Gender Classification Using Ear and Profile Face Images](https://mlsanity.com/paper/1907.10081/) (2019)
- [Controlling for Confounders in Multimodal Emotion Classification via Adversarial Learning](https://mlsanity.com/paper/1908.08979/) (2019)
- [Multimodal Material Classification for Robots using Spectroscopy and High Resolution Texture Imaging](https://mlsanity.com/paper/2004.01160/) (2020)
- [Multimodal price prediction](https://mlsanity.com/paper/2007.05056/) (2020)
- [Hellinger Distance Weighted Ensemble for Imbalanced Data Stream Classification](https://mlsanity.com/paper/2102.00266/) (2021)
- [Multimodal Classification with Deep Convolutional-Recurrent Neural Networks for Electroencephalography](https://mlsanity.com/paper/1807.10641/) (2018)

## Abstract (arXiv)

Multimodal classification research has been gaining popularity in many domains that collect more data from multiple sources including satellite imagery, biometrics, and medicine. However, the lack of consistent terminology and architectural descriptions makes it difficult to compare different existing solutions. We address these challenges by proposing a new taxonomy for describing such systems based on trends found in recent publications on multimodal classification. Many of the most difficult aspects of unimodal classification have not yet been fully addressed for multimodal datasets including big data, class imbalance, and instance level difficulty. We also provide a discussion of these challenges and future directions.