# From Pixels to Torques: Policy Learning with Deep Dynamical Models

arXiv: https://arxiv.org/abs/1502.02251 · HTML: https://mlsanity.com/paper/1502.02251/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2015.

## Built on by (most cited first, 30 of 45)

- [When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learning](https://mlsanity.com/paper/2203.16797/) (2022)
- [Learning Task Informed Abstractions](https://mlsanity.com/paper/2106.15612/) (2021)
- [Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models](https://mlsanity.com/paper/2207.08229/) (2022)
- [DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck](https://mlsanity.com/paper/2102.13268/) (2021)
- [Learning Knowledge Graph-based World Models of Textual Environments](https://mlsanity.com/paper/2106.09608/) (2021)
- [Model-Based Reinforcement Learning via Latent-Space Collocation](https://mlsanity.com/paper/2106.13229/) (2021)
- [On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer](https://mlsanity.com/paper/2312.03673/) (2023)
- [Learning Representations for Pixel-based Control: What Matters and Why?](https://mlsanity.com/paper/2111.07775/) (2021)
- [A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddings](https://mlsanity.com/paper/2405.13245/) (2024)
- [Learning World Models with Identifiable Factorization](https://mlsanity.com/paper/2306.06561/) (2023)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning](https://mlsanity.com/paper/2205.00579/) (2022)
- [Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flight](https://mlsanity.com/paper/2501.14377/) (2025)
- [Latent State Marginalization as a Low-cost Approach for Improving Exploration](https://mlsanity.com/paper/2210.00999/) (2022)
- [On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2210.09256/) (2022)
- [Learning to Optimize in Model Predictive Control](https://mlsanity.com/paper/2212.02603/) (2022)
- [Learning Sampling Distributions for Model Predictive Control](https://mlsanity.com/paper/2212.02587/) (2022)
- [Accelerating Model-Based Reinforcement Learning with State-Space World Models](https://mlsanity.com/paper/2502.20168/) (2025)
- [Do Androids Dream of Electric Fences? Safety-Aware Reinforcement Learning with Latent Shielding](https://mlsanity.com/paper/2112.11490/) (2021)
- [Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL](https://mlsanity.com/paper/2302.05342/) (2023)
- [Deep Model Predictive Optimization](https://mlsanity.com/paper/2310.04590/) (2023)
- [Deep Kernel Learning of Dynamical Models from High-Dimensional Noisy Data](https://mlsanity.com/paper/2208.12975/) (2022)
- [DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors](https://mlsanity.com/paper/2409.18330/) (2024)
- [Enhancing Tactile-based Reinforcement Learning for Robotic Control](https://mlsanity.com/paper/2510.21609/) (2025)
- [Learning Sequential Latent Variable Models from Multimodal Time Series Data](https://mlsanity.com/paper/2204.10419/) (2022)
- [LOPR: Latent Occupancy PRediction using Generative Models](https://mlsanity.com/paper/2210.01249/) (2022)
- [Learning Generalizable Representations for Reinforcement Learning via Adaptive Meta-learner of Behavioral Similarities](https://mlsanity.com/paper/2212.13088/) (2022)
- [Co-Learning Empirical Games and World Models](https://mlsanity.com/paper/2305.14223/) (2023)
- [Automatic Music Playlist Generation via Simulation-based Reinforcement Learning](https://mlsanity.com/paper/2310.09123/) (2023)
- [Learning Multiple Initial Solutions to Optimization Problems](https://mlsanity.com/paper/2411.02158/) (2024)

## Abstract (arXiv)

Data-efficient learning in continuous state-action spaces using very high-dimensional observations remains a key challenge in developing fully autonomous systems. In this paper, we consider one instance of this challenge, the pixels to torques problem, where an agent must learn a closed-loop control policy from pixel information only. We introduce a data-efficient, model-based reinforcement learning algorithm that learns such a closed-loop policy directly from pixel information. The key ingredient is a deep dynamical model that uses deep auto-encoders to learn a low-dimensional embedding of images jointly with a predictive model in this low-dimensional feature space. Joint learning ensures that not only static but also dynamic properties of the data are accounted for. This is crucial for long-term predictions, which lie at the core of the adaptive model predictive control strategy that we use for closed-loop control. Compared to state-of-the-art reinforcement learning methods for continuous states and actions, our approach learns quickly, scales to high-dimensional state spaces and is an important step toward fully autonomous learning from pixels to torques.