# VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing, Speaking, and Acting

arXiv: https://arxiv.org/abs/2510.21817 · HTML: https://mlsanity.com/paper/2510.21817/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 32 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [PaliGemma: A versatile 3B VLM for transfer](https://mlsanity.com/paper/2407.07726/) (2024)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation](https://mlsanity.com/paper/2312.13139/) (2023)
- [Robotic Control via Embodied Chain-of-Thought Reasoning](https://mlsanity.com/paper/2407.08693/) (2024)
- [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025)
- [RT-H: Action Hierarchies Using Language](https://mlsanity.com/paper/2403.01823/) (2024)
- [VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction](https://mlsanity.com/paper/2501.01957/) (2025)
- [VITA: Towards Open-Source Interactive Omni Multimodal LLM](https://mlsanity.com/paper/2408.05211/) (2024)
- [Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning](https://mlsanity.com/paper/2311.17842/) (2023)
- [ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning](https://mlsanity.com/paper/2507.16815/) (2025)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoning](https://mlsanity.com/paper/2505.11917/) (2025)
- [RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning](https://mlsanity.com/paper/2409.14674/) (2024)
- [From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation](https://mlsanity.com/paper/2505.08548/) (2025)
- [SwitchVLA: Execution-Aware Task Switching for Vision-Language-Action Models](https://mlsanity.com/paper/2506.03574/) (2025)
- [Concerns and Values in Human-Robot Interactions: A Focus on Social Robotics](https://mlsanity.com/paper/2501.05628/) (2025)

## Built on by (most cited first, 1 of 1)

- [LLMs are General Asynchronous Agents](https://mlsanity.com/paper/2609.35427/) (2026)

## Abstract (arXiv)

Current Vision-Language-Action (VLA) models are often constrained by a rigid, static interaction paradigm, which lacks the ability to see, hear, speak, and act concurrently as well as handle real-time user interruptions dynamically. This hinders seamless embodied collaboration, resulting in an inflexible and unresponsive user experience. To address these limitations, we introduce VITA-E, a novel embodied interaction framework designed for both behavioral concurrency and nearly real-time interruption. The core of our approach is a dual-model architecture where two parallel VLA instances operate as an ``Active Model'' and a ``Standby Model'', allowing the embodied agent to observe its environment, listen to user speech, provide verbal responses, and execute actions, all concurrently and interruptibly, mimicking human-like multitasking capabilities. We further propose a ``model-as-controller'' paradigm, where we fine-tune the VLM to generate special tokens that serve as direct system-level commands, coupling the model's reasoning with the system's behavior. Experiments conducted on a physical humanoid platform demonstrate that VITA-E can reliably handle complex interactive scenarios. Our framework is compatible with various dual-system VLA models, achieving an extremely high success rate on emergency stops and speech interruptions while also successfully performing concurrent speech and action. This represents a significant step towards more natural and capable embodied assistants.