mlsanityData to 7 Oct 2026

Efficient Large-Scale Multi-Modal Classification

arXiv 1802.02892 ↗First posted 6 Feb 2018cs.CL cs.AI cs.CVcited by 57

Not on the map. Why?

Abstract (arXiv)

While the incipient internet was largely text-based, the modern digital world is becoming increasingly multi-modal. Here, we examine multi-modal classification where one modality is discrete, e.g. text, and the other is continuous, e.g. visual representations transferred from a convolutional neural network. In particular, we focus on scenarios where we have to be able to classify large quantities of data quickly. We investigate various methods for performing multi-modal fusion and analyze their trade-offs in terms of classification accuracy and computational efficiency. Our findings indicate that the inclusion of continuous information improves performance over text-only on a range of multi-modal classification tasks, even with simple fusion methods. In addition, we experiment with discretizing the continuous features in order to speed up and simplify the fusion process even further. Our results show that fusion with discretized features outperforms text-only classification, at a fraction of the computational cost of full multi-modal fusion, with the additional benefit of improved interpretability.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 8; 2023: 10; 2024: 12; 2025: 12; 2026: 3.

57 papers.

Sort:

By area:

  1. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  2. 2022Trusted Multi-View Classification with Dynamic Evidential Fusioncited by 167
  3. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 114
  4. 2021Trusted Multi-View Classificationcited by 94
  5. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 74
  6. 2021Trustworthy Long-Tailed Classificationcited by 51
  7. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  8. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 43
  9. 2021Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributionscited by 39
  10. 2021AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognitioncited by 36
  11. 2021Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognitioncited by 35
  12. 2024Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Modelcited by 30
38 more
  1. 2021Can audio-visual integration strengthen robustness under multimodal attacks?cited by 23
  2. 2022Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysiscited by 22
  3. 2025Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisitioncited by 18
  4. 2024CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusioncited by 17
  5. 2024Improving Multimodal Learning with Multi-Loss Gradient Modulationcited by 16
  6. 2025DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learningcited by 16
  7. 2025Learning Optimal Multimodal Information Bottleneck Representationscited by 15
  8. 2021Decoding EEG Brain Activity for Multi-Modal Natural Language Processingcited by 13
  9. 2023Single-branch Network for Multimodal Trainingcited by 13
  10. 2023FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibrationcited by 13
  11. 2021Audio-Visual Transformer Based Crowd Countingcited by 12
  12. 2023Learning Weakly Supervised Audio-Visual Violence Detection in Hyperbolic Spacecited by 12
  13. 2022RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapycited by 10
  14. 2024Mitigating Noisy Correspondence by Geometrical Structure Consistency Learningcited by 10
  15. 2024Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understandingcited by 10
  16. 2021Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approachcited by 9
  17. 2025Improving Multimodal Learning Balance and Sufficiency through Data Remixingcited by 8
  18. 2024CrisisKAN: Knowledge-infused and Explainable Multimodal Attention Network for Crisis Event Classificationcited by 6
  19. 2022Multi-Modal Attribute Extraction for E-Commercecited by 5
  20. 2025G2D: Boosting Multimodal Learning with Gradient-Guided Distillationcited by 5
  21. 2023Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text Classificationcited by 4
  22. 2024Knockout: A simple way to handle missing inputscited by 4
  23. 2024Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approachcited by 4
  24. 2021A Comparative Study of Speaker Role Identification in Air Traffic Communication Using Deep Learning Approachescited by 3
  25. 2023PATCorrect: Non-autoregressive Phoneme-augmented Transformer for ASR Error Correctioncited by 3
  26. 2021On-Device Document Classification using multimodal featurescited by 2
  27. 2023Modular Multimodal Machine Learning for Extraction of Theorems and Proofs in Long Scientific Documents (Extended Version)cited by 2
  28. 2024Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalitiescited by 2
  29. 2024Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classificationcited by 2
  30. 2025Robust Multimodal Learning via Cross-Modal Proxy Tokenscited by 2
  31. 2026Multimodal Classification via Total Correlation Maximizationcited by 2
  32. 2022DM2S2: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attentioncited by 1
  33. 2022Embeddings as Epistemic States: Limitations on the Use of Pooling Operators for Accumulating Knowledgecited by 1
  34. 2023Balanced Multi-modal Federated Learning via Cross-Modal Infiltrationcited by 1
  35. 2024AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversationscited by 1
  36. 2025Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completioncited by 1
  37. 2025eMotions: A Large-Scale Dataset and Audio-Visual Fusion Network for Emotion Analysis in Short-form Videoscited by 1
  38. 2025Exploring the Synergy of Quantitative Factors and Newsflow Representations from Large Language Models for Stock Return Predictioncited by 1

50 most cited of 57 All 57 (JSON)