# Second-Order MPC-Based Distributed Q-Learning

arXiv: https://arxiv.org/abs/2511.16424 · HTML: https://mlsanity.com/paper/2511.16424/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for autonomous driving and traffic, one of 15 directions in Deep reinforcement learning methods.
- Builds on 4 of its 17 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [RL for autonomous driving and traffic](https://mlsanity.com/deeprl/rl-for-autonomous-driving-and-traffic/)

## Builds on (most cited first, 4 of 4)

- [Data-driven Economic NMPC using Reinforcement Learning](https://mlsanity.com/paper/1904.04152/) (2019)
- [Reinforcement Learning-based Model Predictive Control for Greenhouse Climate Control](https://mlsanity.com/paper/2409.12789/) (2024)
- [Reinforcement Learning with Model Predictive Control for Highway Ramp Metering](https://mlsanity.com/paper/2311.08820/) (2023)
- [Multi-Agent Reinforcement Learning via Distributed MPC as a Function Approximator](https://mlsanity.com/paper/2312.05166/) (2023)

## Abstract (arXiv)

The state of the art for model predictive control (MPC)-based distributed Q-learning is limited to first-order gradient updates of the MPC parameterization. In general, using secondorder information can significantly improve the speed of convergence for learning, allowing the use of higher learning rates without introducing instability. This work presents a second-order extension to MPC-based Q-learning with updates distributed across local agents, relying only on locally available information and neighbor-to-neighbor communication. In simulation the approach is demonstrated to significantly outperform first-order distributed Q-learning.