mlsanityData to 14 Oct 2026

Towards deployment-centric multimodal AI beyond vision and language

arXiv 2504.03603 ↗First posted 4 Apr 2025cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Multimodal artificial intelligence (AI) integrates diverse types of data via machine learning to improve understanding, prediction, and decision-making across disciplines such as healthcare, science, and engineering. However, most multimodal AI advances focus on models for vision and language data, while their deployability remains a key challenge. We advocate a deployment-centric workflow that incorporates deployment constraints early to reduce the likelihood of undeployable solutions, complementing data-centric and model-centric approaches. We also emphasise deeper integration across multiple levels of multimodality and multidisciplinary collaboration to significantly broaden the research scope beyond vision and language. To facilitate this approach, we identify common multimodal-AI-specific challenges shared across disciplines and examine three real-world use cases: pandemic response, self-driving car design, and climate change adaptation, drawing expertise from healthcare, social science, engineering, science, sustainability, and finance. By fostering multidisciplinary dialogue and open research practices, our community can accelerate deployment-centric development for broad societal impact.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 57 of its 161 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

57 of 161 references.

References · 57

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  10. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  11. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  12. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
38 more
  1. 2022Multimodal Learning with Transformers: A Surveycited by 400
  2. 2019Interpretable machine learning: definitions, methods, and applicationscited by 305
  3. 2022DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detectioncited by 273
  4. 2020Neurosymbolic AI: The 3rd Wavecited by 255
  5. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  6. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
  7. 2023WeatherBench 2: A benchmark for the next generation of data-driven global weather modelscited by 236
  8. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 226
  9. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  10. 2024An Introduction to Vision-Language Modelingcited by 178
  11. 2023FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Leadcited by 175
  12. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  13. 2020Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scanscited by 158
  14. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 153
  15. 2020Challenges in Deploying Machine Learning: a Survey of Case Studiescited by 152
  16. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  17. 2019Self-Driving Cars: A Surveycited by 148
  18. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  19. 2022Machine Learning Operations (MLOps): Overview, Definition, and Architecturecited by 128
  20. 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
  21. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  22. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  23. 2022A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodalcited by 106
  24. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 106
  25. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 101
  26. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  27. 2018Privacy Preserving Machine Learning: Threats and Solutionscited by 79
  28. 2024Scaling Wearable Foundation Modelscited by 63
  29. 2023FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcarecited by 60
  30. 2020Multi-fidelity Bayesian Neural Networks: Algorithms and Applicationscited by 59
  31. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  32. 2024Design Principles for Generative AI Applicationscited by 57
  33. 2022Multimodal learning with graphscited by 54
  34. 2024ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Changecited by 47
  35. 2018Machine Learning for Integrating Data in Biology and Medicine: Principles, Practice, and Opportunitiescited by 43
  36. 2024Review of multimodal machine learning approaches in healthcarecited by 31
  37. 2020Technology Readiness Levels for AI & MLcited by 30
  38. 2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Bias-constrained multimodal intelligence for equitable and reliable clinical AIcited by 0
  2. 2026Prediction certification cannot replace explanation certification: a competence envelope for trustworthy AI under compound stresscited by 0
  3. 2026Towards Privacy-Preserving Thermal Human Perception from Dataset to Deploymentcited by 0