mlsanityData to 14 Oct 2026

VLASCD: A Visual Language Action Model for Simultaneous Chatting and Decision Making

arXiv 2410.15885 ↗First posted 21 Oct 2024cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent large pretrained models such as LLMs (e.g., GPT series) and VLAs (e.g., OpenVLA) have achieved notable progress on multimodal tasks, yet they are built upon a multi-input single-output (MISO) paradigm. We show that this paradigm fundamentally limits performance in multi-input multi-output (MIMO) scenarios, where parallel task execution is required. In MISO architectures, tasks compete for a shared output channel, creating mutual exclusion effects that cause unbalanced optimization and degraded performance. To address this gap, we introduce MIMO-VLA (VLASCD), a unified training framework that enables concurrent multi-task outputs, exemplified by simultaneous dialogue generation and decision-making. Inspired by human cognition, MIMO-VLA eliminates interference between tasks and supports efficient parallel processing. Experiments on the CARLA autonomous driving platform demonstrate that MIMO-VLA substantially outperforms state-of-the-art MISO-based LLMs, reinforcement learning models, and VLAs in MIMO settings, establishing a new direction for multimodal and multitask learning.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

29 of 46 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  8. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
17 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  3. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  4. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  5. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 402
  6. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  7. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
  8. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 232
  9. 2023Is ChatGPT a Good Recommender? A Preliminary Studycited by 193
  10. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  11. 2019Multimodal End-to-End Autonomous Drivingcited by 110
  12. 2019Model-free Deep Reinforcement Learning for Urban Autonomous Drivingcited by 98
  13. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 58
  14. 2022Enhance Sample Efficiency and Robustness of End-to-end Urban Autonomous Driving via Semantic Masked World Modelcited by 45
  15. 2022Flow-based Recurrent Belief State Learning for POMDPscited by 16
  16. 2024Efficient and Generalized end-to-end Autonomous Driving System with Latent Deep Reinforcement Learning and Demonstrationscited by 7
  17. 2024A Framework to Implement 1+N Multi-task Fine-tuning Pattern in LLMs Using the CGC-LORA Algorithmcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
  2. 2025VLAs are Confined yet Capable of Generalizing to Novel Instructions
  3. 2023Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation
  4. 2026The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning
  5. 2025MM-ACT: Learn from Multimodal Parallel Generation to Act
  6. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models
  7. 2026UAM: A Dual-Stream Perspective on Forgetting in VLA Training
  8. 2024A Survey on Vision-Language-Action Models for Embodied AI