# Physically Interpretable World Models via Weakly Supervised Representation Learning

arXiv: https://arxiv.org/abs/2412.12870 · HTML: https://mlsanity.com/paper/2412.12870/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Latent predictive world models, one of 11 directions in World models.
- Builds on 46 of its 71 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Placed in

- World models / [Latent predictive world models](https://mlsanity.com/wm/latent-predictive-world-models/)

## Builds on (most cited first, 30 of 46)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Transformers are Sample-Efficient World Models](https://mlsanity.com/paper/2209.00588/) (2022)
- [OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving](https://mlsanity.com/paper/2311.16038/) (2023)
- [Any-point Trajectory Modeling for Policy Learning](https://mlsanity.com/paper/2401.00025/) (2023)
- [Masked World Models for Visual Control](https://mlsanity.com/paper/2206.14244/) (2022)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction](https://mlsanity.com/paper/2412.10373/) (2024)
- [Learning Task Informed Abstractions](https://mlsanity.com/paper/2106.15612/) (2021)
- [RenderWorld: World Model with Self-Supervised 3D Label](https://mlsanity.com/paper/2409.11356/) (2024)
- [VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning](https://mlsanity.com/paper/2410.23156/) (2024)
- [Conformal Prediction for STL Runtime Verification](https://mlsanity.com/paper/2211.01539/) (2022)
- [UniWorld: Autonomous Driving Pre-training via World Models](https://mlsanity.com/paper/2308.07234/) (2023)
- [Verification of Image-based Neural Network Controllers Using Generative Models](https://mlsanity.com/paper/2105.07091/) (2021)
- [Generative ODE Modeling with Known Unknowns](https://mlsanity.com/paper/2003.10775/) (2020)
- [SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels](https://mlsanity.com/paper/2410.08822/) (2024)
- [Marrying Causal Representation Learning with Dynamical Systems for Science](https://mlsanity.com/paper/2405.13888/) (2024)
- [Interpretable Self-Aware Neural Networks for Robust Trajectory Prediction](https://mlsanity.com/paper/2211.08701/) (2022)
- [PI-VAE: Physics-Informed Variational Auto-Encoder for stochastic differential equations](https://mlsanity.com/paper/2203.11363/) (2022)
- [FOCUS: Object-Centric World Models for Robotics Manipulation](https://mlsanity.com/paper/2307.02427/) (2023)

## Built on by (most cited first, 3 of 3)

- [Four Principles for Physically Interpretable World Models](https://mlsanity.com/paper/2503.02143/) (2025)
- [World Model for Battery Degradation Prediction Under Non-Stationary Aging](https://mlsanity.com/paper/2603.10527/) (2026)
- [Stochastic World Models for Verifying Vision-Based Neural Feedback Systems](https://mlsanity.com/paper/2609.38120/) (2026)

## Abstract (arXiv)

Learning predictive models from high-dimensional sensory observations is fundamental for cyber-physical systems, yet the latent representations learned by standard world models lack physical interpretability. This limits their reliability, generalizability, and applicability to safety-critical tasks. We introduce Physically Interpretable World Models (PIWM), a framework that aligns latent representations with real-world physical quantities and constrains their evolution through partially known physical dynamics. Physical interpretability in PIWM is defined by two complementary properties: (i) the learned latent state corresponds to meaningful physical variables, and (ii) its temporal evolution follows physically consistent dynamics. To achieve this without requiring ground-truth physical annotations, PIWM employs weak distribution-based supervision that captures state uncertainty naturally arising from real-world sensing pipelines. The architecture integrates a VQ-based visual encoder, a transformer-based physical encoder, and a learnable dynamics model grounded in known physical equations. Across three case studies (Cart Pole, Lunar Lander, and Donkey Car), PIWM achieves accurate long-horizon prediction, recovers true system parameters, and significantly improves physical grounding over purely data-driven models. These results demonstrate the feasibility and advantages of learning physically interpretable world models directly from images under weak supervision.