World Modeling through Spectral Alignment
On the map: World models → Latent predictive world models
Abstract (arXiv)
How do we tell a world model what information to preserve? We study this question through the lens of spectral representation learning. We introduce SpecWM, a JEPA framework that explicitly specifies desired relationships between observations through a target similarity kernel and trains the encoder to reproduce those relationships in its embedding geometry. Different kernels provide different supervision, ranging from physical state information to a fully self-supervised temporal kernel requiring only the ordering of observations within trajectories. We characterize the global optima of the objective, establish conditions for linear recoverability and invariance to nuisance information, and connect temporal-kernel similarity to the behavioral successor measure under an idealized setting. Empirically, the choice of kernel strongly controls which physical quantities are recoverable from the representation. More importantly, temporal spectral alignment improves CEM planning success over LeWorldModel from 52% to 68% on OGBench Cube, 49% to 77% on OGBench Scene, and 29% to 51% on CALVIN. These results suggest that explicitly specifying representation geometry provides a simple way to steer what latent world models preserve for control.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 41 of its 51 references that are in the map. First posted on arXiv 11 Oct 2026.
Builds on
41 of 51 references.
Closest ancestors · 4
Same direction, newest first
- 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels13 Mar
- 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning7 Nov
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture19 Jan
- 2018Learning Latent Dynamics for Planning from Pixels12 Nov
Other references · 37
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2018World Modelscited by 1,513
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
- 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
- 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
- 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
- 2026World Action Models are Zero-shot Policiescited by 465
25 more
- 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
- 2026Causal World Modeling for Robot Controlcited by 321
- 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
- 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
- 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 300
- 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
- 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
- 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
- 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
- 2025Training Agents Inside of Scalable World Modelscited by 136
- 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
- 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
- 2021Learning One Representation to Optimize All Rewardscited by 115
- 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
- 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
- 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 68
- 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 44
- 2026HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worldscited by 44
- 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 42
- 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
- 2022Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddingscited by 24
- 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 22
- 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 12
- 2018The Deep Kernelized Autoencodercited by 3
- 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Spectral-Target Physical Latent Structuring for JEPA-Style World Models
- 2026SCALE: State-Calibrated Latent Embeddings for JEPA Planning in the Right Geometry
- 2026Platonic Representation Hypothesis on World Models
- 2026Emergent Semantic Representations in World Models through Physical Interaction without Linguistic Supervision
- 2026ATLAS: Aligned Transport of Latent Structure for Reliable World Model Planning
- 2026LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
- 2026Flow Equivariant World Models: Memory for Partially Observed Dynamic Environments
- 2026Geometric Priors for Generalizable World Models via Vector Symbolic Architecture