# Reinforcement Learning with Formal Performance Metrics for Quadcopter Attitude Control under Non-nominal Contexts

arXiv: https://arxiv.org/abs/2107.12942 · HTML: https://mlsanity.com/paper/2107.12942/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 76 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guarantees](https://mlsanity.com/paper/1909.05304/) (2019)
- [Deep Drone Acrobatics](https://mlsanity.com/paper/2006.05768/) (2020)
- [Q-Learning for Robust Satisfaction of Signal Temporal Logic Specifications](https://mlsanity.com/paper/1609.07409/) (2016)
- [Reinforcement Learning for UAV Attitude Control](https://mlsanity.com/paper/1804.04154/) (2018)
- [Sim-to-(Multi)-Real: Transfer of Low-Level Robust Control Policies to Multiple Quadrotors](https://mlsanity.com/paper/1903.04628/) (2019)
- [Reinforcement Learning with Non-Markovian Rewards](https://mlsanity.com/paper/1912.02552/) (2019)
- [Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/1901.03737/) (2019)
- [D2RL: Deep Dense Architectures in Reinforcement Learning](https://mlsanity.com/paper/2010.09163/) (2020)
- [Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning](https://mlsanity.com/paper/2010.04304/) (2020)
- [Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functions](https://mlsanity.com/paper/1903.09885/) (2019)
- [A Smooth Robustness Measure of Signal Temporal Logic for Symbolic Control](https://mlsanity.com/paper/2006.05239/) (2020)
- [MAMPS: Safe Multi-Agent Reinforcement Learning via Model Predictive Shielding](https://mlsanity.com/paper/1910.12639/) (2019)
- [Arithmetic-Geometric Mean Robustness for Control from Signal Temporal Logic Specifications](https://mlsanity.com/paper/1903.05186/) (2019)
- [Control from Signal Temporal Logic Specifications with Smooth Cumulative Quantitative Semantics](https://mlsanity.com/paper/1904.11611/) (2019)
- [Correct-by-synthesis reinforcement learning with temporal logic constraints](https://mlsanity.com/paper/1503.01793/) (2015)
- [AReN: Assured ReLU NN Architecture for Model Predictive Control of LTI Systems](https://mlsanity.com/paper/1911.01608/) (2019)
- [Time Robustness in MTL and Expressivity in Hybrid System Falsification (Extended Version)](https://mlsanity.com/paper/1505.06307/) (2015)
- [How to Make Deep RL Work in Practice](https://mlsanity.com/paper/2010.13083/) (2020)
- [Design of a Trajectory Tracking Controller for a Nanoquadcopter](https://mlsanity.com/paper/1608.05786/) (2016)
- [Two-Level Lattice Neural Network Architectures for Control of Nonlinear Systems](https://mlsanity.com/paper/2004.09628/) (2020)
- [Multi-Armed Bandits for Boolean Connectives in Hybrid System Falsification (Extended Version)](https://mlsanity.com/paper/1905.07549/) (2019)
- [Safe Reinforcement Learning with Nonlinear Dynamics via Model Predictive Shielding](https://mlsanity.com/paper/1905.10691/) (2019)

## Built on by (most cited first, 3 of 3)

- [Prescribing Optimal Health-Aware Operation for Urban Air Mobility with Deep Reinforcement Learning](https://mlsanity.com/paper/2404.08497/) (2024)
- [cc-DRL: a Convex Combined Deep Reinforcement Learning Flight Control Design for a Morphing Quadrotor](https://mlsanity.com/paper/2408.13054/) (2024)
- [A Heuristic Approach for Performance Tuning in RL-based Quadrotor Control via Reward Design and Termination Conditions](https://mlsanity.com/paper/2605.19166/) (2026)

## Abstract (arXiv)

We explore the reinforcement learning approach to designing controllers by extensively discussing the case of a quadcopter attitude controller. We provide all details allowing to reproduce our approach, starting with a model of the dynamics of a crazyflie 2.0 under various nominal and non-nominal conditions, including partial motor failures and wind gusts. We develop a robust form of a signal temporal logic to quantitatively evaluate the vehicle's behavior and measure the performance of controllers. The paper thoroughly describes the choices in training algorithms, neural net architecture, hyperparameters, observation space in view of the different performance metrics we have introduced. We discuss the robustness of the obtained controllers, both to partial loss of power for one rotor and to wind gusts and finish by drawing conclusions on practical controller design by reinforcement learning.