Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds
On the map: World models → Video-generation world models and simulators
Abstract (arXiv)
Video generation is progressing beyond isolated clips toward long-form narratives and interactive worlds, requiring models to preserve identities, follow user controls, and remain stable over extended rollouts. We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants. The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio, enabling persistent character appearance and voice identity across flexible combinations of text, image, and memory conditioning. The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway, enabling controller-agnostic interaction across flexible viewpoints. To support efficient long-horizon generation, we transform a bidirectional audio-visual backbone into a causal few-step generator using progressive teacher forcing and short- and long-horizon Self-Gradient Forcing on self-generated rollouts. Experiments demonstrate strong performance in both settings. JoyAI-Echo-1.5 achieves improvements over existing long-video baselines in cross-shot consistency, visual quality, text alignment, and speech fidelity. Its world-model variant ranks first on WBench, with an average score of 81.7, and achieves leading visual quality and long-horizon persistence on SANA-WM-Bench. Together, these results indicate that memory, geometric control, and rollout-aware training provide a practical foundation for generating coherent stories and continuously evolving interactive worlds. Project page: https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 48 of its 53 references that are in the map. First posted on arXiv 24 Aug 2026.
Builds on
48 of 53 references.
Closest ancestors · 14
Same direction, newest first
- 2026Alaya-EVOKE: From Linear-Scaling Supervision to Endless World13 Aug
- 2026ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU21 Jul
- 2026DreamX-World 1.0: A General-Purpose Interactive World Model15 Jun
- 2026Cosmos 3: Omnimodal World Models for Physical AI1 Jun
- 2026WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation25 May
- 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer14 May
- 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory10 Apr
- 2026Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memory2 Feb
- 2026Advancing Open-source World Models28 Jan
- 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling16 Dec
- 2025Unified Camera Positional Encoding for Controlled Video Generation8 Dec
- 2025Wan: Open and Advanced Large-Scale Video Generative Models26 Mar
2 more
Other references · 34
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
- 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
- 2023Consistency Modelscited by 1,822
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
- 2023One-step Diffusion with Distribution Matching Distillationcited by 1,019
- 2021Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Datacited by 970
- 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
- 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 684
- 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
22 more
- 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
- 2025Mean Flows for One-step Generative Modelingcited by 561
- 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
- 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 450
- 2026LTX-2: Efficient Joint Audio-Visual Foundation Modelcited by 236
- 2025ViPE: Video Pose Engine for 3D Geometric Perceptioncited by 170
- 2025HunyuanVideo 1.5 Technical Reportcited by 160
- 2021Investigating Tradeoffs in Real-World Video Super-Resolutioncited by 134
- 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
- 2025Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Modelcited by 80
- 2026Lyra 2.0: Explorable Generative 3D Worldscited by 44
- 2019Torchreid: A Library for Deep Learning Person Re-Identification in Pytorchcited by 41
- 2025StoryMem: Multi-shot Long Video Storytelling with Memorycited by 40
- 2024KVQ: Kwai Video Quality Assessment for Short-form Videoscited by 38
- 2026ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytellingcited by 28
- 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 25
- 2026Self Gradient Forcing: Native Long Video Extrapolationcited by 17
- 20243D-Speaker-Toolkit: An Open-Source Toolkit for Multimodal Speaker Verification and Diarizationcited by 16
- 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 16
- 2026Ultra Flash: Scaling Real-Time Streaming Video Generation to High Resolutionscited by 3
- 2026AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generationcited by 1
- 2026OmniVR: Audio-Video Conditional Generation for Archival Footage Restorationcited by 1
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Controlcited by 4
- 2026WorldWeave: Growing Persistent Geometric Worlds for Video Generationcited by 0
- 2026WorldSonus: Bringing Sound to Worldscited by 0
- 2026SGF+: Decoupling Gradient Flows for Autoregressive Video Generationcited by 0
- 2026SGF+: Decoupling Gradient Flows for Autoregressive Video Generation7 Oct 2026
- 2026WorldSonus: Bringing Sound to Worlds6 Oct 2026
- 2026WorldWeave: Growing Persistent Geometric Worlds for Video Generation28 Sep 2026
- 2026Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control15 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2026EchoWM: Open and Enterable Omnimodal World Models
- 2026LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV
- 2026HelixWorld: A Real-time Interactive Audio-Visual World Model
- 2025Controllable Audio-Visual Viewpoint Generation from 360deg Spatial Information
- 2026DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution
- 2026OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars
- 2026Vorch-Omni: Multi-Task Orchestration of Sight and Sound
- 2024Audio-Synchronized Visual Animation