Proactive Physical AI through Continuous Observation and Timely Intervention
Not on the map. Why?
Abstract (arXiv)
Physical AI is commonly formulated around an explicitly specified task: given a goal or instruction, a system perceives, reasons, and acts. In many real-world settings, however, important tasks emerge from continuously changing world states before a person explicitly formulates them. We study this complementary problem as Proactive Physical AI, where an AI system continuously observes the physical world, identifies situations that warrant intervention, and initiates appropriate actions at the right time. We develop ProAgent, an end-to-end system built around two components: a Jev-style proactive timing model (dubbed StreamingJev) that determines whether and when intervention is needed, and a Response Model that generates responses for people or invokes available device capabilities for physical execution. To support learning at scale, we develop an automated data pipeline that converts videos into supervision for proactive timing. Experiments demonstrate strong proactive performance under both explicit objectives and standing roles, combining more accurate intervention timing with fewer false triggers than the source-backbone baseline. Real-robot evaluations further demonstrate ProAgent's ability to identify emerging needs and translate them into appropriate physical actions.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 12 Oct 2026.
Builds on
27 of 32 references.
References · 27
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
- 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
- 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 166
- 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
- 2023RoboVQA: Multimodal Long-Horizon Reasoning for Roboticscited by 120
- 2025Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledgecited by 73
15 more
- 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
- 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 51
- 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 46
- 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 42
- 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 28
- 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 28
- 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 27
- 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 25
- 2024StreamChat: Chatting with Streaming Videocited by 24
- 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 22
- 2025ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wildcited by 10
- 2025RoboOmni: Proactive Robot Manipulation in Omni-modal Contextcited by 8
- 2026ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devicescited by 6
- 2026ProAct: Harnessing Streaming Motion Generation and Agentic Reasoning for Real-Time Embodied Social Interactioncited by 3
- 2026Event-Driven Proactive Assistive Manipulation with Grounded Vision-Language Planningcited by 2
Built on by
Not cited yet.