# Efficient Large-Scale Multi-Modal Classification

arXiv: https://arxiv.org/abs/1802.02892 · HTML: https://mlsanity.com/paper/1802.02892/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2018.

## Built on by (most cited first, 30 of 57)

- [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://mlsanity.com/paper/2211.08332/) (2022)
- [Trusted Multi-View Classification with Dynamic Evidential Fusion](https://mlsanity.com/paper/2204.11423/) (2022)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Trusted Multi-View Classification](https://mlsanity.com/paper/2102.02051/) (2021)
- [Multimodal Representation Learning by Alternating Unimodal Adaptation](https://mlsanity.com/paper/2311.10707/) (2023)
- [Trustworthy Long-Tailed Classification](https://mlsanity.com/paper/2111.09030/) (2021)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Enhancing multimodal cooperation via sample-level modality valuation](https://mlsanity.com/paper/2309.06255/) (2023)
- [Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions](https://mlsanity.com/paper/2111.08456/) (2021)
- [AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition](https://mlsanity.com/paper/2105.05165/) (2021)
- [Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognition](https://mlsanity.com/paper/2110.10101/) (2021)
- [Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model](https://mlsanity.com/paper/2405.18014/) (2024)
- [Can audio-visual integration strengthen robustness under multimodal attacks?](https://mlsanity.com/paper/2104.02000/) (2021)
- [Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysis](https://mlsanity.com/paper/2206.06367/) (2022)
- [Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition](https://mlsanity.com/paper/2503.18595/) (2025)
- [Improving Multimodal Learning with Multi-Loss Gradient Modulation](https://mlsanity.com/paper/2405.07930/) (2024)
- [CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion](https://mlsanity.com/paper/2402.05889/) (2024)
- [DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning](https://mlsanity.com/paper/2503.06456/) (2025)
- [Learning Optimal Multimodal Information Bottleneck Representations](https://mlsanity.com/paper/2505.19996/) (2025)
- [Decoding EEG Brain Activity for Multi-Modal Natural Language Processing](https://mlsanity.com/paper/2102.08655/) (2021)
- [Single-branch Network for Multimodal Training](https://mlsanity.com/paper/2303.06129/) (2023)
- [FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibration](https://mlsanity.com/paper/2307.16617/) (2023)
- [Audio-Visual Transformer Based Crowd Counting](https://mlsanity.com/paper/2109.01926/) (2021)
- [Learning Weakly Supervised Audio-Visual Violence Detection in Hyperbolic Space](https://mlsanity.com/paper/2305.18797/) (2023)
- [RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapy](https://mlsanity.com/paper/2204.12423/) (2022)
- [Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning](https://mlsanity.com/paper/2405.16996/) (2024)
- [Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding](https://mlsanity.com/paper/2407.05910/) (2024)
- [Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approach](https://mlsanity.com/paper/2108.08920/) (2021)
- [Improving Multimodal Learning Balance and Sufficiency through Data Remixing](https://mlsanity.com/paper/2506.11550/) (2025)
- [CrisisKAN: Knowledge-infused and Explainable Multimodal Attention Network for Crisis Event Classification](https://mlsanity.com/paper/2401.06194/) (2024)

## Abstract (arXiv)

While the incipient internet was largely text-based, the modern digital world is becoming increasingly multi-modal. Here, we examine multi-modal classification where one modality is discrete, e.g. text, and the other is continuous, e.g. visual representations transferred from a convolutional neural network. In particular, we focus on scenarios where we have to be able to classify large quantities of data quickly. We investigate various methods for performing multi-modal fusion and analyze their trade-offs in terms of classification accuracy and computational efficiency. Our findings indicate that the inclusion of continuous information improves performance over text-only on a range of multi-modal classification tasks, even with simple fusion methods. In addition, we experiment with discretizing the continuous features in order to speed up and simplify the fusion process even further. Our results show that fusion with discretized features outperforms text-only classification, at a fraction of the computational cost of full multi-modal fusion, with the additional benefit of improved interpretability.