mlsanityData to 7 Oct 2026

HelixWorld: A Real-time Interactive Audio-Visual World Model

arXiv 2609.38123 ↗First posted 29 Sep 2026cs.CV cs.MM cs.SDcited by 0

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

World simulation is inherently multisensory, demanding synchronized visual and acoustic dynamics in real time. Yet prevailing interactive world models remain strictly silent, focusing exclusively on visual rendering and control while overlooking the acoustic dimension. We present HelixWorld, a real-time interactive audio-visual world model where visual scenes and camera-grounded spatial stereo sound co-evolve natively under user interaction. We curate a high-fidelity spatial audio-visual dataset with true stereo acoustics and metric camera poses, upon which we pre-train a bidirectional teacher conditioned on 6-DoF camera trajectories and user actions. To enable low-latency causal interaction, we distill the teacher into a few-step streaming student via an online trajectory distillation loss, sustaining drift-free joint audio-visual rollouts at 24 FPS on a single GPU. Furthermore, we formalize spatial-acoustic consistency and introduce HelixBench to evaluate whether synthesized sound fields faithfully track dynamic viewpoint motion. Extensive experiments demonstrate that HelixWorld matches state-of-the-art silent world models in visual fidelity and responsiveness, while significantly surpassing existing baselines in camera-aligned spatial-acoustic immersion.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 26 of its 56 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

26 of 56 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control15 Sep
  2. 2026EchoWM: Open and Enterable Omnimodal World Models24 Aug
  3. 2026Alaya-EVOKE: From Linear-Scaling Supervision to Endless World13 Aug
  4. 2026WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation25 May
  5. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer14 May
  6. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory10 Apr
  7. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling16 Dec

Other references · 19

  1. 2018World Modelscited by 1,444
  2. 2025DINOv3cited by 1,297
  3. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
  4. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  5. 2025Qwen3-Omni Technical Reportcited by 451
  6. 2025LongLive: Real-time Interactive Long Video Generationcited by 237
  7. 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 213
  8. 2025ViPE: Video Pose Engine for 3D Geometric Perceptioncited by 162
  9. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  10. 2025Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Conditioncited by 92
  11. 2026Infinite Worlds with Versatile Interactionscited by 37
  12. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 36
7 more
  1. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 33
  2. 2026Diversity-Preserved Distribution Matching Distillation for Fast Visual Synthesiscited by 20
  3. 2026VGGT-Omegacited by 20
  4. 2025Matrix-Game: Interactive World Foundation Modelcited by 18
  5. 2026PP-OCRv6: From 1.5M to 34.5M Parameters, Surpassing Billion-Scale VLMs on OCR Taskscited by 12
  6. 2025PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generationcited by 7
  7. 2026DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026WorldSonus: Bringing Sound to Worlds
  2. 2026Audible World Models: Spatially Aware Sound Generation for 3D Worlds
  3. 2025Audio-Visual World Models: Learning Physically Grounded Multisensory Dynamics
  4. 2026Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds
  5. 2026Scene2Sound: Auditory-Grounded Soundscape Generation for 3D Gaussian Worlds
  6. 2025Controllable Audio-Visual Viewpoint Generation from 360deg Spatial Information
  7. 2026SyncCache: Exploiting Asymmetric Dynamics for Fast Audio-Driven Portrait Animation
  8. 2026MaineCoon: Pursuing A Real-Time Audio-Visual Social World Model