mlsanityData to 7 Oct 2026

MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning

arXiv 2306.16413 ↗First posted 28 Jun 2023cs.LG cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiZoo, a public toolkit consisting of standardized implementations of > 20 core multimodal algorithms and MultiBench, a large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. Together, these provide an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, we offer a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench paves the way towards a better understanding of the capabilities and limitations of multimodal models, while ensuring ease of use, accessibility, and reproducibility. Our toolkits are publicly available, will be regularly updated, and welcome inputs from the community.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 29 of its 48 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

29 of 48 references.

References · 29

  1. 2017The Kinetics Human Action Video Datasetcited by 1,894
  2. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  3. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  4. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  5. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  6. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  7. 2017Gated Multimodal Units for Information Fusioncited by 244
  8. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  9. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 193
  10. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
  11. 2018Learning Factorized Multimodal Representationscited by 171
  12. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
17 more
  1. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  2. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  3. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
  4. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  5. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  6. 2019MFAS: Multimodal Fusion Architecture Searchcited by 84
  7. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
  8. 2020Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropiescited by 56
  9. 2022Dynamic Multimodal Fusioncited by 54
  10. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 45
  11. 2018Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learningcited by 43
  12. 2018Multimodal Machine Learning for Automated ICD Codingcited by 40
  13. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  14. 2022High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learningcited by 31
  15. 2019Learning Representations from Imperfect Time Series Data via Tensor Rank Regularizationcited by 30
  16. 2020Multimodal Sensor Fusion with Differentiable Filterscited by 29
  17. 2021Multimodal Fusion Refiner Networkscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025A Principled Framework for Multi-View Contrastive Learningcited by 4
  2. 2024eipy: An Open-Source Python Package for Multi-modal Data Integration using Heterogeneous Ensemblescited by 0
  3. 2025No Data Wasted: A Semi-supervised Generative Model for Incomplete Multi-view Data Integration with Missing Labelscited by 0
  4. 2026MuteBench: Modality Unavailability Tolerance Evaluation for Incomplete Multimodal Fusioncited by 0