# Training Efficient Controllers via Analytic Policy Gradient

arXiv: https://arxiv.org/abs/2209.13052 · HTML: https://mlsanity.com/paper/2209.13052/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 26 Sep 2022.

## Builds on (most cited first, 30 of 30)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [Learning to Control PDEs with Differentiable Physics](https://mlsanity.com/paper/2001.07457/) (2020)
- [A Differentiable Physics Engine for Deep Learning in Robotics](https://mlsanity.com/paper/1611.01652/) (2016)
- [Model-Based Reinforcement Learning via Meta-Policy Optimization](https://mlsanity.com/paper/1809.05214/) (2018)
- [gradSim: Differentiable simulation for system identification and visuomotor control](https://mlsanity.com/paper/2104.02646/) (2021)
- [Gradients are Not All You Need](https://mlsanity.com/paper/2111.05803/) (2021)
- [Flightmare: A Flexible Quadrotor Simulator](https://mlsanity.com/paper/2009.00563/) (2020)
- [A Differentiable Programming System to Bridge Machine Learning and Scientific Computing](https://mlsanity.com/paper/1907.07587/) (2019)
- [Scalable Differentiable Physics for Learning and Control](https://mlsanity.com/paper/2007.02168/) (2020)
- [Autonomous Drone Racing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.08624/) (2021)
- [Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework](https://mlsanity.com/paper/1912.12970/) (2019)
- [MBRL-Lib: A Modular Library for Model-based Reinforcement Learning](https://mlsanity.com/paper/2104.10159/) (2021)
- [Differential Flatness of Quadrotor Dynamics Subject to Rotor Drag for Accurate Tracking of High-Speed Trajectories](https://mlsanity.com/paper/1712.02402/) (2017)
- [One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priors](https://mlsanity.com/paper/1509.06841/) (2015)
- [Performance, Precision, and Payloads: Adaptive Nonlinear MPC for Quadrotors](https://mlsanity.com/paper/2109.04210/) (2021)
- [Learning Quadrotor Dynamics Using Neural Network for Flight Control](https://mlsanity.com/paper/1610.05863/) (2016)
- [Deluca -- A Differentiable Control Library: Environments, Methods, and Benchmarking](https://mlsanity.com/paper/2102.09968/) (2021)
- [Total stochastic gradient algorithms and applications in reinforcement learning](https://mlsanity.com/paper/1902.01722/) (2019)
- [Leveraging Reward Gradients For Reinforcement Learning in Differentiable Physics Simulations](https://mlsanity.com/paper/2203.02857/) (2022)
- [Full-Pose Tracking Control for Aerial Robotic Systems with Laterally-Bounded Input Force](https://mlsanity.com/paper/1605.06645/) (2016)
- [Soft Multicopter Control using Neural Dynamics Identification](https://mlsanity.com/paper/2008.07689/) (2020)

## Built on by (most cited first, 22 of 22)

- [Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation](https://mlsanity.com/paper/2410.03076/) (2024)
- [SkyJEPA: Learning Long-Horizon World Models for Zero-Shot Sim-to-Real Control of Quadrotors](https://mlsanity.com/paper/2606.23444/) (2026)
- [VisFly: An Efficient and Versatile Simulator for Training Vision-based Flight](https://mlsanity.com/paper/2407.14783/) (2024)
- [Accelerating Visual-Policy Learning through Parallel Differentiable Simulation](https://mlsanity.com/paper/2505.10646/) (2025)
- [Behavior Optimized Image Generation](https://mlsanity.com/paper/2311.10995/) (2023)
- [DiffSim2Real: Deploying Quadrupedal Locomotion Policies Purely Trained in Differentiable Simulation](https://mlsanity.com/paper/2411.02189/) (2024)
- [R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks](https://mlsanity.com/paper/2504.01250/) (2025)
- [Deep Reinforcement Learning for Sequential Combinatorial Auctions](https://mlsanity.com/paper/2407.08022/) (2024)
- [Autonomous Vehicle Controllers From End-to-End Differentiable Simulation](https://mlsanity.com/paper/2409.07965/) (2024)
- [Unlocking Efficient Vehicle Dynamics Modeling via Analytic World Models](https://mlsanity.com/paper/2502.10012/) (2025)
- [React to Surprises: Stable-by-Design Neural Feedback Control and the Youla-REN](https://mlsanity.com/paper/2506.01226/) (2025)
- [Autonomous Vehicle Path Planning by Searching With Differentiable Simulation](https://mlsanity.com/paper/2511.11043/) (2025)
- [ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards](https://mlsanity.com/paper/2501.14513/) (2025)
- [Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control](https://mlsanity.com/paper/2512.24826/) (2025)
- [VisFly-Lab: Unified Differentiable Framework for First-Order Reinforcement Learning of Quadrotor Control](https://mlsanity.com/paper/2603.21123/) (2026)
- [Tempered Sequential Monte Carlo for Trajectory and Policy Optimization with Differentiable Dynamics](https://mlsanity.com/paper/2604.21456/) (2026)
- [ELMP: Efficient Learning for Motion Planning via Analytical Policy Gradients](https://mlsanity.com/paper/2607.00215/) (2026)
- [Learning Agile Intruder Interception using Differentiable Quadrotor Dynamics](https://mlsanity.com/paper/2607.02472/) (2026)
- [Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems](https://mlsanity.com/paper/2607.16177/) (2026)
- [Accelerating Visual Policy Learning with Sampling-Based Model Predictive Control](https://mlsanity.com/paper/2609.20575/) (2026)
- [Learning Vision-Based Agile Gap Traversal: Differentiable Simulation with a Warm-Started Critic](https://mlsanity.com/paper/2609.30696/) (2026)
- [When Does Backpropagating Through Policy Memory Matter? Physical Credit, Optimizer Updates, and Observability](https://mlsanity.com/paper/2609.33169/) (2026)

## Abstract (arXiv)

Control design for robotic systems is complex and often requires solving an optimization to follow a trajectory accurately. Online optimization approaches like Model Predictive Control (MPC) have been shown to achieve great tracking performance, but require high computing power. Conversely, learning-based offline optimization approaches, such as Reinforcement Learning (RL), allow fast and efficient execution on the robot but hardly match the accuracy of MPC in trajectory tracking tasks. In systems with limited compute, such as aerial vehicles, an accurate controller that is efficient at execution time is imperative. We propose an Analytic Policy Gradient (APG) method to tackle this problem. APG exploits the availability of differentiable simulators by training a controller offline with gradient descent on the tracking error. We address training instabilities that frequently occur with APG through curriculum learning and experiment on a widely used controls benchmark, the CartPole, and two common aerial robots, a quadrotor and a fixed-wing drone. Our proposed method outperforms both model-based and model-free RL methods in terms of tracking error. Concurrently, it achieves similar performance to MPC while requiring more than an order of magnitude less computation time. Our work provides insights into the potential of APG as a promising control method for robotics. To facilitate the exploration of APG, we open-source our code and make it available at https://github.com/lis-epfl/apg_trajectory_tracking.