# Learning Policies for Continuous Control via Transition Models

arXiv: https://arxiv.org/abs/2209.08033 · HTML: https://mlsanity.com/paper/2209.08033/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 13 of 13)

- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Deep Active Inference as Variational Policy Gradients](https://mlsanity.com/paper/1907.03876/) (2019)
- [Active Inference in Robotics and Artificial Agents: Survey and Challenges](https://mlsanity.com/paper/2112.01871/) (2021)
- [Deep Active Inference](https://mlsanity.com/paper/1709.02341/) (2017)
- [Whence the Expected Free Energy?](https://mlsanity.com/paper/2004.08128/) (2020)
- [Hybrid Predictive Coding: Inferring, Fast and Slow](https://mlsanity.com/paper/2204.02169/) (2022)
- [End-to-End Pixel-Based Deep Active Inference for Body Perception and Action](https://mlsanity.com/paper/2001.05847/) (2019)
- [Deep Active Inference for Partially Observable MDPs](https://mlsanity.com/paper/2009.03622/) (2020)
- [Learning Perception and Planning with Deep Active Inference](https://mlsanity.com/paper/2001.11841/) (2020)
- [Bayesian policy selection using active inference](https://mlsanity.com/paper/1904.08149/) (2019)
- [Control as Hybrid Inference](https://mlsanity.com/paper/2007.05838/) (2020)
- [Adaptation through prediction: multisensory active inference torque control](https://mlsanity.com/paper/2112.06752/) (2021)

## Built on by (most cited first, 3 of 3)

- [Information-seeking polynomial NARX model-predictive control through expected free energy minimization](https://mlsanity.com/paper/2312.15046/) (2023)
- [Coupled autoregressive active inference agents for control of multi-joint dynamical systems](https://mlsanity.com/paper/2410.10415/) (2024)
- [Spiking Neural Networks for Continuous Control via End-to-End Model-Based Learning](https://mlsanity.com/paper/2509.05356/) (2025)

## Abstract (arXiv)

It is doubtful that animals have perfect inverse models of their limbs (e.g., what muscle contraction must be applied to every joint to reach a particular location in space). However, in robot control, moving an arm's end-effector to a target position or along a target trajectory requires accurate forward and inverse models. Here we show that by learning the transition (forward) model from interaction, we can use it to drive the learning of an amortized policy. Hence, we revisit policy optimization in relation to the deep active inference framework and describe a modular neural network architecture that simultaneously learns the system dynamics from prediction errors and the stochastic policy that generates suitable continuous control commands to reach a desired reference position. We evaluated the model by comparing it against the baseline of a linear quadratic regulator, and conclude with additional steps to take toward human-like motor control.