# Model-free Learning with Heterogeneous Dynamical Systems: A Federated LQR Approach

arXiv: https://arxiv.org/abs/2308.11743 · HTML: https://mlsanity.com/paper/2308.11743/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2023.

## Built on by (most cited first, 23 of 23)

- [Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning](https://mlsanity.com/paper/2401.15273/) (2024)
- [Meta-Learning Linear Quadratic Regulators: A Policy Gradient MAML Approach for Model-free LQR](https://mlsanity.com/paper/2401.14534/) (2024)
- [Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments](https://mlsanity.com/paper/2405.19499/) (2024)
- [Nonasymptotic Regret Analysis of Adaptive Linear Quadratic Control with Model Misspecification](https://mlsanity.com/paper/2401.00073/) (2023)
- [Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries](https://mlsanity.com/paper/2403.09940/) (2024)
- [Coreset-Based Task Selection for Sample-Efficient Meta-Reinforcement Learning](https://mlsanity.com/paper/2502.02332/) (2025)
- [Policy Gradient for LQR with Domain Randomization](https://mlsanity.com/paper/2503.24371/) (2025)
- [Asynchronous Heterogeneous Linear Quadratic Regulator Design](https://mlsanity.com/paper/2404.09061/) (2024)
- [Regret Analysis of Multi-task Representation Learning for Linear-Quadratic Adaptive Control](https://mlsanity.com/paper/2407.05781/) (2024)
- [A survey on secure decentralized optimization and learning](https://mlsanity.com/paper/2408.08628/) (2024)
- [Combining Federated Learning and Control: A Survey](https://mlsanity.com/paper/2407.11069/) (2024)
- [Learning Stabilizing Policies via an Unstable Subspace Representation](https://mlsanity.com/paper/2505.01348/) (2025)
- [Oracle Complexity Reduction for Model-free LQR: A Stochastic Variance-Reduced Policy Gradient Approach](https://mlsanity.com/paper/2309.10679/) (2023)
- [Model-Free Learning for the Linear Quadratic Regulator over Rate-Limited Channels](https://mlsanity.com/paper/2401.01258/) (2024)
- [A Moreau Envelope Approach for LQR Meta-Policy Estimation](https://mlsanity.com/paper/2403.17364/) (2024)
- [Policy Gradient Bounds in Multitask LQR](https://mlsanity.com/paper/2509.19266/) (2025)
- [FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivative](https://mlsanity.com/paper/2512.20814/) (2025)
- [Asynchronous Parallel Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/2407.03233/) (2024)
- [Adversarially Robust Multitask Adaptive Control](https://mlsanity.com/paper/2511.05444/) (2025)
- [Harnessing Data from Clustered LQR Systems: Personalized and Collaborative Policy Optimization](https://mlsanity.com/paper/2511.17489/) (2025)
- [Multitask LQG Control: Performance and Generalization Bounds](https://mlsanity.com/paper/2604.16730/) (2026)
- [FedScalar: Federated Learning with Scalar Communication for Bandwidth-Constrained Networks](https://mlsanity.com/paper/2410.02260/) (2024)
- [Scalar Federated Learning for Linear Quadratic Regulator](https://mlsanity.com/paper/2604.05088/) (2026)

## Abstract (arXiv)

We study a model-free federated linear quadratic regulator (LQR) problem where M agents with unknown, distinct yet similar dynamics collaboratively learn an optimal policy to minimize an average quadratic cost while keeping their data private. To exploit the similarity of the agents' dynamics, we propose to use federated learning (FL) to allow the agents to periodically communicate with a central server to train policies by leveraging a larger dataset from all the agents. With this setup, we seek to understand the following questions: (i) Is the learned common policy stabilizing for all agents? (ii) How close is the learned common policy to each agent's own optimal policy? (iii) Can each agent learn its own optimal policy faster by leveraging data from all agents? To answer these questions, we propose a federated and model-free algorithm named FedLQR. Our analysis overcomes numerous technical challenges, such as heterogeneity in the agents' dynamics, multiple local updates, and stability concerns. We show that FedLQR produces a common policy that, at each iteration, is stabilizing for all agents. We provide bounds on the distance between the common policy and each agent's local optimal policy. Furthermore, we prove that when learning each agent's optimal policy, FedLQR achieves a sample complexity reduction proportional to the number of agents M in a low-heterogeneity regime, compared to the single-agent setting.