# EnerVerse-AC: Envisioning Embodied Environments with Action Condition arXiv: https://arxiv.org/abs/2505.09723 ยท HTML: https://mlsanity.com/paper/2505.09723/ ## Key facts - Cited by 42 papers in the map (data through 14 Oct 2026). - Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. - Builds on 25 of its 34 references that are in the map. - First posted on arXiv 14 May 2025. ## Placed in - Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/) ## Builds on (most cited first, 25 of 25) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022) - [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024) - [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022) - [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024) - [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023) - [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021) - [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024) - [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018) - [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024) - [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019) - [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023) - [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023) - [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023) - [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023) - [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023) - [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024) - [AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems](https://mlsanity.com/paper/2503.06669/) (2025) - [Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos](https://mlsanity.com/paper/2304.01186/) (2023) - [DisCo: Disentangled Control for Realistic Human Dance Generation](https://mlsanity.com/paper/2307.00040/) (2023) - [Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want](https://mlsanity.com/paper/2403.20271/) (2024) - [Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation](https://mlsanity.com/paper/2410.08001/) (2024) - [Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learning](https://mlsanity.com/paper/2503.11646/) (2025) ## Built on by (most cited first, 30 of 42) - [Ctrl-World: A Controllable Generative World Model for Robot Manipulation](https://mlsanity.com/paper/2510.10125/) (2025) - [DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos](https://mlsanity.com/paper/2602.06949/) (2026) - [WorldEval: World Model as Real-World Robot Policies Evaluator](https://mlsanity.com/paper/2505.19017/) (2025) - [GigaWorld-0: World Models as Data Engine to Empower Embodied AI](https://mlsanity.com/paper/2511.19861/) (2025) - [WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL](https://mlsanity.com/paper/2602.13977/) (2026) - [World Model for Robot Learning: A Comprehensive Survey](https://mlsanity.com/paper/2605.00080/) (2026) - [GigaBrain-0: A World Model-Powered Vision-Language-Action Model](https://mlsanity.com/paper/2510.19430/) (2025) - [ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignment](https://mlsanity.com/paper/2603.23376/) (2026) - [Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models](https://mlsanity.com/paper/2603.25716/) (2026) - [GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learning](https://mlsanity.com/paper/2602.12099/) (2026) - [Act2Goal: From World Model To General Goal-conditioned Policy](https://mlsanity.com/paper/2512.23541/) (2025) - [BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Masks](https://mlsanity.com/paper/2602.03793/) (2026) - [FlowWAM: Optical Flow as a Unified Action Representation for World Action Models](https://mlsanity.com/paper/2607.13017/) (2026) - [Co-Evolving Latent Action World Models](https://mlsanity.com/paper/2510.26433/) (2025) - [Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test](https://mlsanity.com/paper/2601.04137/) (2026) - [Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation](https://mlsanity.com/paper/2603.16669/) (2026) - [ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation](https://mlsanity.com/paper/2603.27670/) (2026) - [AdaWorldPolicy: World-Model-Driven Diffusion Policy with Online Adaptive Learning for Robotic Manipulation](https://mlsanity.com/paper/2602.20057/) (2026) - [Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation](https://mlsanity.com/paper/2606.18960/) (2026) - [LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE](https://mlsanity.com/paper/2509.21790/) (2025) - [Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos](https://mlsanity.com/paper/2511.12882/) (2025) - [RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL](https://mlsanity.com/paper/2512.03556/) (2025) - [Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms](https://mlsanity.com/paper/2603.28489/) (2026) - [A Definition and Roadmap for World Models](https://mlsanity.com/paper/2607.06401/) (2026) - [GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation](https://mlsanity.com/paper/2609.05588/) (2026) - [FreeAction: Training-Free Techniques for Enhanced Fidelity of Trajectory-to-Video Generation](https://mlsanity.com/paper/2509.24241/) (2025) - [Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control Interface](https://mlsanity.com/paper/2512.19402/) (2025) - [PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation](https://mlsanity.com/paper/2606.18375/) (2026) - [RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation](https://mlsanity.com/paper/2607.01060/) (2026) - [Envisioning the Future, One Step at a Time](https://mlsanity.com/paper/2604.09527/) (2026) ## Abstract (arXiv) Robotic imitation learning has advanced from solving static tasks to addressing dynamic interaction scenarios, but testing and evaluation remain costly and challenging due to the need for real-time interaction with dynamic environments. We propose EnerVerse-AC (EVAC), an action-conditional world model that generates future visual observations based on an agent's predicted actions, enabling realistic and controllable robotic inference. Building on prior architectures, EVAC introduces a multi-level action-conditioning mechanism and ray map encoding for dynamic multi-view image generation while expanding training data with diverse failure trajectories to improve generalization. As both a data engine and evaluator, EVAC augments human-collected trajectories into diverse datasets and generates realistic, action-conditioned video observations for policy testing, eliminating the need for physical robots or complex simulations. This approach significantly reduces costs while maintaining high fidelity in robotic manipulation evaluation. Extensive experiments validate the effectiveness of our method. Code, checkpoints, and datasets can be found at .