Formal Policy Learning from Demonstrations for Reachability Properties
On the map: Offline RL and imitation learning from data → Robot learning from demonstration
Abstract (arXiv)
We consider the problem of learning structured, closed-loop policies (feedback laws) from demonstrations in order to control under-actuated robotic systems, so that formal behavioral specifications such as reaching a target set of states are satisfied. Our approach uses a ``counterexample-guided'' iterative loop that involves the interaction between a policy learner, a demonstrator and a verifier. The learner is responsible for querying the demonstrator in order to obtain the training data to guide the construction of a policy candidate. This candidate is analyzed by the verifier and either accepted as correct, or rejected with a counterexample. In the latter case, the counterexample is used to update the training data and further refine the policy. The approach is instantiated using receding horizon model-predictive controllers (MPCs) as demonstrators. Rather than using regression to fit a policy to the demonstrator actions, we extend the MPC formulation with the gradient of the cost-to-go function evaluated at sample states in order to constrain the set of policies compatible with the behavior of the demonstrator. We demonstrate the successful application of the resulting policy learning schemes on two case studies and we show how simple, formally-verified policies can be inferred starting from a complex and unverified nonlinear MPC implementations. As a further benefit, the policies are many orders of magnitude faster to implement when compared to the original MPCs.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data. Builds on 6 of its 34 references that are in the map. First posted on arXiv 2 Mar 2019.
Builds on
6 of 34 references.
References · 6
- 2016PLATO: Policy Learning using Adaptive Trajectory Optimizationcited by 24
- 2018Learning Control Lyapunov Functions from Counterexamples and Demonstrationscited by 19
- 2015A Theory of Formal Synthesis via Inductive Learningcited by 17
- 2015Counter-Example Guided Synthesis of Control Lyapunov Functions for Switched Systemscited by 4
- 2015Controller Synthesis with Inductive Proofs for Piecewise Linear Systems: an SMT-based Algorithmcited by 3
- 2016Sound and Automated Synthesis of Digital Stabilizing Controllers for Continuous Plantscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2018Constraint Estimation and Derivative-Free Recovery for Robot Learning from Demonstrations
- 2024Towards Improving Learning from Demonstration Algorithms via MCMC Methods
- 2019MPC-Net: A First Principles Guided Policy Search
- 2026Learning from Demonstrations meets Symbolic Control - A Supervisory Control Architecture for Formal Certification of Learned Motion
- 2026Learning from Imperfect Demonstrations via Temporal Behavior Tree-Guided Trajectory Repair
- 2025Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach
- 2026RFPO: Rectified Flow Policy Optimization for Embodied Control
- 2018Differentiable MPC for End-to-end Planning and Control