# Model-Based Reinforcement Learning via Stochastic Hybrid Models

arXiv: https://arxiv.org/abs/2111.06211 · HTML: https://mlsanity.com/paper/2111.06211/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 92 references that are in the map.
- First posted on arXiv 11 Nov 2021.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Switching Linear Dynamics for Variational Bayes Filtering](https://mlsanity.com/paper/1905.12434/) (2019)
- [f-Divergence constrained policy improvement](https://mlsanity.com/paper/1801.00056/) (2017)
- [Adaptive Skills, Adaptive Partitions (ASAP)](https://mlsanity.com/paper/1602.03351/) (2016)
- [A Bayesian Approach to Policy Recognition and State Representation Learning](https://mlsanity.com/paper/1605.01278/) (2016)
- [Hybrid system identification using switching density networks](https://mlsanity.com/paper/1907.04360/) (2019)
- [Hierarchical Decomposition of Nonlinear Dynamics and Control for System Identification and Policy Distillation](https://mlsanity.com/paper/2005.01432/) (2020)

## Built on by (most cited first, 2 of 2)

- [Hybrid Recurrent Models Support Emergent Descriptions for Hierarchical Planning and Control](https://mlsanity.com/paper/2408.10970/) (2024)
- [Learning in Hybrid Active Inference Models](https://mlsanity.com/paper/2409.01066/) (2024)

## Abstract (arXiv)

Optimal control of general nonlinear systems is a central challenge in automation. Enabled by powerful function approximators, data-driven approaches to control have recently successfully tackled challenging applications. However, such methods often obscure the structure of dynamics and control behind black-box over-parameterized representations, thus limiting our ability to understand closed-loop behavior. This paper adopts a hybrid-system view of nonlinear modeling and control that lends an explicit hierarchical structure to the problem and breaks down complex dynamics into simpler localized units. We consider a sequence modeling paradigm that captures the temporal structure of the data and derive an expectation-maximization (EM) algorithm that automatically decomposes nonlinear dynamics into stochastic piecewise affine models with nonlinear transition boundaries. Furthermore, we show that these time-series models naturally admit a closed-loop extension that we use to extract local polynomial feedback controllers from nonlinear experts via behavioral cloning. Finally, we introduce a novel hybrid relative entropy policy search (Hb-REPS) technique that incorporates the hierarchical nature of hybrid models and optimizes a set of time-invariant piecewise feedback controllers derived from a piecewise polynomial approximation of a global state-value function.