# Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models

arXiv: https://arxiv.org/abs/2605.06388 · HTML: https://mlsanity.com/paper/2605.06388/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 68 of its 80 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)
- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 68)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)

## Built on by (most cited first, 22 of 22)

- [LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies](https://mlsanity.com/paper/2606.15768/) (2026)
- [EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data](https://mlsanity.com/paper/2607.08436/) (2026)
- [Making Foresight Actionable: Repurposing Representation Alignment in World Action Models](https://mlsanity.com/paper/2606.12217/) (2026)
- [What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction](https://mlsanity.com/paper/2606.07687/) (2026)
- [Kairos: A Regret-Aware Native World-Action Model Stack for Physical AI](https://mlsanity.com/paper/2606.16533/) (2026)
- [Is Forward Prediction Enough? Physical State Grounding for JEPA World Models](https://mlsanity.com/paper/2608.06799/) (2026)
- [Nano World Models: A Minimalist Implementation of Future Video Prediction](https://mlsanity.com/paper/2605.23993/) (2026)
- [ContactWorld: What Representations Matter for Vision-Tactile Latent World Models in Contact-Rich Manipulation](https://mlsanity.com/paper/2606.13877/) (2026)
- [Robust-WAM: Bridging Generative Pretraining and Semantic Foresight in World-Action Models](https://mlsanity.com/paper/2608.05903/) (2026)
- [HALO-WA: Hybrid-Attention Latent-Guided Online Reinforcement Learning for World-Action Models](https://mlsanity.com/paper/2607.04265/) (2026)
- [Rethinking Representations for World-Action Modeling](https://mlsanity.com/paper/2609.38163/) (2026)
- [Decoupling Intention from Trajectory: A Representational Deduction Framework for World Action Models](https://mlsanity.com/paper/2608.06994/) (2026)
- [SegDem: Segmentation helps Demosaicing](https://mlsanity.com/paper/2608.07916/) (2026)
- [SLIM-0.5B: Learning Action-Grounded Predictive Latents for Robot Manipulation](https://mlsanity.com/paper/2608.09771/) (2026)
- [SlotDiT: Object-Centric Representations for Diffusion Transformers](https://mlsanity.com/paper/2609.17414/) (2026)
- [MT-WAM: Reorienting the One-Pass Predictive Representation Toward Action Generation](https://mlsanity.com/paper/2609.21474/) (2026)
- [Visual Representation and History Modeling for Navigation World Models](https://mlsanity.com/paper/2609.29555/) (2026)
- [Reimagine Video Dynamics](https://mlsanity.com/paper/2609.36496/) (2026)
- [The Domain Is a Residue: Adapting Self-Supervised Features, Not Generators](https://mlsanity.com/paper/2609.37330/) (2026)
- [CogWAM: Aligning Semantic Cognition with World Action Modeling via Event-Driven Interfaces](https://mlsanity.com/paper/2609.37721/) (2026)
- [H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning](https://mlsanity.com/paper/2610.06805/) (2026)
- [RoboJEPA: Scaling Robotic Latent World Models](https://mlsanity.com/paper/2610.10515/) (2026)

## Abstract (arXiv)

World model-based policy evaluation is a practical proxy for testing real-world robot control by rolling out candidate actions in action-conditioned video diffusion models. As these models increasingly adopt latent diffusion modeling (LDM), choosing the right latent space becomes critical. While the status quo uses autoencoding latent spaces like VAEs that are primarily trained for pixel reconstruction, recent work suggests benefits from pretrained encoders with representation-aligned semantic latent spaces. We systematically evaluate these latent spaces for action-conditioned LDM by comparing six reconstruction and semantic encoders to train world model variants under a fixed protocol on BridgeV2 dataset, and show effective world model training in high-dimensional representation spaces with and without dimension compression. We then propose three axes to assess robotic world model performance: visual fidelity, planning and downstream policy performance, and latent representation quality. Our results show visual fidelity alone is insufficient for world model selection. While reconstruction encoders like VAE and Cosmos achieve strong pixel-level scores, semantic encoders such as V-JEPA 2.1 (strongest overall on policy), Web-DINO, and SigLIP 2 generally excel across the other two axes at all model scales. Our study advocates semantic latent space as stronger foundation for policy-relevant robotics diffusion world models.