What Visual Representations Do World Action Models Need?
On the map: World models → Embodied and robot world models also fits: Video-generation world models and simulators
Abstract (arXiv)
World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. External visual representations can improve manipulation, but existing methods couple representation choices with different integration mechanisms, obscuring their individual effects. We systematically compare visual representations, five integration mechanisms, and feature combinations under controlled settings. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert's effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R2-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 46 of its 51 references that are in the map. First posted on arXiv 9 Oct 2026.
Builds on
46 of 51 references.
Closest ancestors · 15
Same direction, newest first
- 2026Latent Action as Intention Enables Efficient Future Imagination for World Action Models25 Aug
- 20264D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields8 Aug
- 2026Robust-WAM: Bridging Generative Pretraining and Semantic Foresight in World-Action Models6 Aug
- 2026ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts31 Jul
- 2026Learning 4D Geometric Priors for Inference-Efficient World Action Models6 Jul
- 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies14 Jun
- 2026RepWAM: World Action Modeling with Representation Visual-Action Tokenizers11 Jun
- 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Models10 Jun
- 2026GeoSem-WAM: Geometry- and Semantic-Aware World Action Models2 Jun
- 2026DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA31 Mar
- 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?17 Mar
- 2026World Action Models are Zero-shot Policies17 Feb
3 more
Other references · 31
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
- 2025DINOv3cited by 1,414
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
- 2025SAM 3: Segment Anything with Conceptscited by 906
- 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
19 more
- 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
- 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 518
- 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
- 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
- 2019Robust Learning with Jacobian Regularizationcited by 129
- 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
- 2026DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Controlcited by 76
- 2026LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestioncited by 52
- 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulationcited by 50
- 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 44
- 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 36
- 2026StarVLA-α: Reducing Complexity in Vision-Language-Action Systemscited by 31
- 2026C-RADIOv4 (Tech Report)cited by 12
- 2026Flex-π: A Multi-Stream World-Action Model with Compute Flexibilitycited by 11
- 2026Vision Pretraining for Dense Spatial Perceptioncited by 8
- 2026DreamWorld: Unified World Modeling in Video Generationcited by 7
- 2023Identifying Important Sensory Feedback for Learning Locomotion Skillscited by 6
- 2026DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representationcited by 5
- 2026WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videoscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Latent evolving World Action Model
- 2026What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction
- 2026The Right Future for Action: Learning Action-Relevant Predictive States in World Action Models
- 2026ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?
- 2026V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
- 2025WorldVLA: Towards Autoregressive Action World Model
- 2026InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data
- 2026World Tokens: Enhancing Embodied Policies with Training-Time World Modeling