mlsanityData to 7 Oct 2026

MUTEX: Learning Unified Policies from Multimodal Task Specifications

arXiv 2309.14320 ↗First posted 25 Sep 2023cs.ROcited by 70

Not on the map. Why?

Cited by 25 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Humans use different modalities, such as speech, text, images, videos, etc., to communicate their intent and goals with teammates. For robots to become better assistants, we aim to endow them with the ability to follow instructions and understand tasks specified by their human partners. Most robotic policy learning methods have focused on one single modality of task specification while ignoring the rich cross-modal information. We present MUTEX, a unified approach to policy learning from multimodal task specifications. It trains a transformer-based architecture to facilitate cross-modal reasoning, combining masked modeling and cross-modal matching objectives in a two-stage training procedure. After training, MUTEX can follow a task specification in any of the six learned modalities (video demonstrations, goal images, text goal descriptions, text instructions, speech goal descriptions, and speech instructions) or a combination of them. We systematically evaluate the benefits of MUTEX in a newly designed dataset with 100 tasks in simulation and 50 tasks in the real world, annotated with multiple instances of task specifications in different modalities, and observe improved performance over methods trained specifically for any single modality. More information at https://ut-austin-rpl.github.io/MUTEX/

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Sep 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 31; 2025: 26; 2026: 10.

70 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  4. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  5. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  6. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  7. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  8. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  9. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  10. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  11. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  12. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
38 more
  1. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  2. 2025Unified Vision-Language-Action Modelcited by 128
  3. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  4. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  5. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  6. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  7. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  8. 2023LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discoverycited by 63
  9. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  10. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  11. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructionscited by 50
  12. 2025VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulationcited by 48
  13. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  14. 2024Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Taskscited by 44
  15. 2024Diffusion Transformer Policycited by 41
  16. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  17. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
  18. 2024Multi-Task Interactive Robot Fleet Learning with Visual World Modelscited by 33
  19. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  20. 2024An Interactive Agent Foundation Modelcited by 25
  21. 2024Natural Language Can Help Bridge the Sim2Real Gapcited by 21
  22. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 19
  23. 2024Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitationcited by 18
  24. 2024PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulationcited by 17
  25. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 17
  26. 2023Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuningcited by 15
  27. 2024On the Vulnerability of LLM/VLM-Controlled Roboticscited by 15
  28. 2025XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representationscited by 15
  29. 2025Efficient Robotic Policy Learning via Latent Space Backward Planningcited by 14
  30. 2024MaIL: Improving Imitation Learning with Mambacited by 11
  31. 2025Adaptive Perception for Unified Visual Multi-modal Object Trackingcited by 11
  32. 2025TwinVLA: Data-Efficient Bimanual Manipulation with Twin Single-Arm Vision-Language-Action Modelscited by 10
  33. 2025HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policiescited by 10
  34. 2025OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learningcited by 10
  35. 2026MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generationcited by 10
  36. 2025SCIZOR: A Self-Supervised Approach to Data Curation for Large-Scale Imitation Learningcited by 9
  37. 2024MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planningcited by 8
  38. 2026Chain of World: World Model Thinking in Latent Motioncited by 8

50 most cited of 70 All 70 (JSON)