# Coordinating Planning and Tracking in Layered Control Policies via Actor-Critic Learning

arXiv: https://arxiv.org/abs/2408.01639 · HTML: https://mlsanity.com/paper/2408.01639/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 3 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 17 of 17)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Near-Optimal Representation Learning for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1810.01257/) (2018)
- [Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator](https://mlsanity.com/paper/1712.08642/) (2017)
- [CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimality](https://mlsanity.com/paper/2211.06625/) (2022)
- [Unified Multi-Rate Control: from Low Level Actuation to High Level Planning](https://mlsanity.com/paper/2012.06558/) (2020)
- [Actor-Critic Physics-informed Neural Lyapunov Control](https://mlsanity.com/paper/2403.08448/) (2024)
- [Towards a Theory of Control Architecture: A quantitative framework for layered multi-rate control](https://mlsanity.com/paper/2401.15185/) (2024)
- [Multi-Rate Planning and Control of Uncertain Nonlinear Systems: Model Predictive Control and Control Lyapunov Functions](https://mlsanity.com/paper/2204.00152/) (2022)
- [Augmented Lagrangian Methods as Layered Control Architectures](https://mlsanity.com/paper/2311.06404/) (2023)
- [A Data-Driven Approach to Synthesizing Dynamics-Aware Trajectories for Underactuated Robotic Systems](https://mlsanity.com/paper/2307.13782/) (2023)
- [Why Change Your Controller When You Can Change Your Planner: Drag-Aware Trajectory Generation for Quadrotor Systems](https://mlsanity.com/paper/2401.04960/) (2024)

## Abstract (arXiv)

We propose a reinforcement learning (RL)-based algorithm to jointly train (1) a trajectory planner and (2) a tracking controller in a layered control architecture. Our algorithm arises naturally from a rewrite of the underlying optimal control problem that lends itself to an actor-critic learning approach. By explicitly learning a textitdual network to coordinate the interaction between the planning and tracking layers, we demonstrate the ability to achieve an effective consensus between the two components, leading to an interpretable policy. We theoretically prove that our algorithm converges to the optimal dual network in the Linear Quadratic Regulator (LQR) setting and empirically validate its applicability to nonlinear systems through simulation experiments on a unicycle model.