mlsanityData to 7 Oct 2026

A Vision for Multisensory Intelligence: Sensing, Science, and Synergy

arXiv 2601.04563 ↗First posted 8 Jan 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Our experience of the world is multisensory, spanning a synthesis of language, sight, sound, touch, taste, and smell. Yet, artificial intelligence has primarily advanced in digital modalities like text, vision, and audio. This paper outlines a research vision for multisensory artificial intelligence over the next decade. This new set of technologies can change how humans and AI experience and interact with one another, by connecting AI to the human senses and a rich spectrum of signals from physiological and tactile cues on the body, to physical and social signals in homes, cities, and the environment. We outline how this field must advance through three interrelated themes of sensing, science, and synergy. Firstly, research in sensing should extend how AI captures the world in richer ways beyond the digital medium. Secondly, developing a principled science for quantifying multimodal heterogeneity and interactions, developing unified modeling architectures and representations, and understanding cross-modal transfer. Finally, we present new technical challenges to learn synergy between modalities and between humans and AI, covering multisensory integration, alignment, reasoning, generation, generalization, and experience. Accompanying this vision paper are a series of projects, resources, and demos of latest advances from the Multisensory Intelligence group at the MIT Media Lab, see https://mit-mi.github.io/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 263 of its 403 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

263 of 403 references.

References · 263

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  12. 2023Visual Instruction Tuningcited by 7,792
38 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  13. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  14. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  15. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  16. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  17. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  18. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  19. 2022Video Diffusion Modelscited by 2,280
  20. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  21. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  22. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  23. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  24. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  25. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  26. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  27. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  28. 2019Contrastive Multiview Codingcited by 1,297
  29. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  30. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  31. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  32. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  33. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  34. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  35. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  36. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  37. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  38. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973

50 most cited of 263 All 263 (JSON)

Built on by

Not cited yet.