# Model-Based Planning with Discrete and Continuous Actions

arXiv: https://arxiv.org/abs/1705.07177 · HTML: https://mlsanity.com/paper/1705.07177/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2017.

## Built on by (most cited first, 7 of 7)

- [Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning](https://mlsanity.com/paper/2010.12142/) (2020)
- [Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2210.07636/) (2022)
- [Slot-MPC: Goal-Conditioned Model Predictive Control with Object-Centric Representations](https://mlsanity.com/paper/2605.14937/) (2026)
- [Cloth Manipulation Planning on Basis of Mesh Representations with Incomplete Domain Knowledge and Voxel-to-Mesh Estimation](https://mlsanity.com/paper/2103.08137/) (2021)
- [Beyond Myopic World Models: Long-Horizon End-to-End Training for Direct Future Prediction](https://mlsanity.com/paper/2608.07420/) (2026)
- [Visual Goal-Directed Meta-Learning with Contextual Planning Networks](https://mlsanity.com/paper/2111.09908/) (2021)
- [Visual processing in context of reinforcement learning](https://mlsanity.com/paper/2208.12525/) (2022)

## Abstract (arXiv)

Action planning using learned and differentiable forward models of the world is a general approach which has a number of desirable properties, including improved sample complexity over model-free RL methods, reuse of learned models across different tasks, and the ability to perform efficient gradient-based optimization in continuous action spaces. However, this approach does not apply straightforwardly when the action space is discrete. In this work, we show that it is in fact possible to effectively perform planning via backprop in discrete action spaces, using a simple paramaterization of the actions vectors on the simplex combined with input noise when training the forward model. Our experiments show that this approach can match or outperform model-free RL and discrete planning methods on gridworld navigation tasks in terms of performance and/or planning time while using limited environment interactions, and can additionally be used to perform model-based control in a challenging new task where the action space combines discrete and continuous actions. We furthermore propose a policy distillation approach which yields a fast policy network which can be used at inference time, removing the need for an iterative planning procedure.