# Assured Learning-enabled Autonomy: A Metacognitive Reinforcement Learning Framework

arXiv: https://arxiv.org/abs/2103.12558 · HTML: https://mlsanity.com/paper/2103.12558/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 64 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Builds on (most cited first, 14 of 14)

- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Reinforcement Learning With Temporal Logic Rewards](https://mlsanity.com/paper/1612.03471/) (2016)
- [Safe Controller Optimization for Quadrotors with Gaussian Processes](https://mlsanity.com/paper/1509.01066/) (2015)
- [A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specifications](https://mlsanity.com/paper/1409.5486/) (2014)
- [A Policy Search Method For Temporal Logic Specified Reinforcement Learning Tasks](https://mlsanity.com/paper/1709.09611/) (2017)
- [Barrier-Certified Adaptive Reinforcement Learning with Applications to Brushbot Navigation](https://mlsanity.com/paper/1801.09627/) (2018)
- [Correct-by-synthesis reinforcement learning with temporal logic constraints](https://mlsanity.com/paper/1503.01793/) (2015)
- [A trajectory-based framework for data-driven system analysis and control](https://mlsanity.com/paper/1903.10723/) (2019)
- [Combining system identification with reinforcement learning-based MPC](https://mlsanity.com/paper/2004.03265/) (2020)
- [Brain Intelligence: Go Beyond Artificial Intelligence](https://mlsanity.com/paper/1706.01040/) (2017)
- [Data-driven Dynamic Multi-objective Optimal Control: An Aspiration-satisfying Reinforcement Learning Approach](https://mlsanity.com/paper/2005.07118/) (2020)
- [Robust Control for Signal Temporal Logic Specifications using Average Space Robustness](https://mlsanity.com/paper/1607.07019/) (2016)
- [Timed Automata Approach for Motion Planning Using Metric Interval Temporal Logic](https://mlsanity.com/paper/1603.08246/) (2016)
- [Adaptive Dynamic Programming for Model-free Tracking of Trajectories with Time-varying Parameters](https://mlsanity.com/paper/1909.07239/) (2019)

## Built on by (most cited first, 3 of 3)

- [Metacognition is all you need? Using Introspection in Generative Agents to Improve Goal-directed Behavior](https://mlsanity.com/paper/2401.10910/) (2024)
- [Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learning](https://mlsanity.com/paper/2204.07729/) (2022)
- [A Risk-Averse Preview-based Q-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles](https://mlsanity.com/paper/2112.03232/) (2021)

## Abstract (arXiv)

Reinforcement learning (RL) agents with pre-specified reward functions cannot provide guaranteed safety across variety of circumstances that an uncertain system might encounter. To guarantee performance while assuring satisfaction of safety constraints across variety of circumstances, an assured autonomous control framework is presented in this paper by empowering RL algorithms with metacognitive learning capabilities. More specifically, adapting the reward function parameters of the RL agent is performed in a metacognitive decision-making layer to assure the feasibility of RL agent. That is, to assure that the learned policy by the RL agent satisfies safety constraints specified by signal temporal logic while achieving as much performance as possible. The metacognitive layer monitors any possible future safety violation under the actions of the RL agent and employs a higher-layer Bayesian RL algorithm to proactively adapt the reward function for the lower-layer RL agent. To minimize the higher-layer Bayesian RL intervention, a fitness function is leveraged by the metacognitive layer as a metric to evaluate success of the lower-layer RL agent in satisfaction of safety and liveness specifications, and the higher-layer Bayesian RL intervenes only if there is a risk of lower-layer RL failure. Finally, a simulation example is provided to validate the effectiveness of the proposed approach.