mlsanityData to 14 Oct 2026

Distributed TD Tracking with Linear Function Approximation over Directed Communication Networks

arXiv 2605.04466 ↗First posted 6 May 2026math.OCcited by 0

On the map: Deep reinforcement learning methods → Multi-agent reinforcement learning

Abstract (arXiv)

We study the policy evaluation problem in multi-agent reinforcement learning (MARL) over directed communication networks, where agents cooperate with each other to explore an unknown environment and accomplish a specific task. We propose a Push-Pull-type distributed algorithm, named PP-DTD, for policy evaluation in MARL within the framework of temporal difference (TD) learning with linear function approximation. PP-DTD integrates TD learning with the Push-Pull mechanism to accommodate directed communication networks, and further utilizes variance reduction techniques to enhance both algorithmic stability and convergence rate. We show that PP-DTD achieves linear convergence to a neighborhood of the optimum under constant step-sizes and a convergence rate of O(T-1) under decaying step-sizes when the sample is independent and identically distributed or Markovian. To the best of our knowledge, PP-DTD is the first distributed algorithm for policy evaluation in MARL over directed graphs that achieves a comparable convergence rate to single-agent TD. The numerical experiments on cooperative navigation tasks demonstrate the robustness and effectiveness of PP-DTD.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multi-agent reinforcement learning, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 25 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

11 of 25 references.

References · 11

  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  4. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  5. 2016Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphscited by 150
  6. 2018Push-Pull Gradient Methods for Distributed Optimization in Networkscited by 46
  7. 2018Deep Reinforcement Learning for Swarm Systemscited by 45
  8. 2019Finite-Time Performance of Distributed Temporal Difference Learning with Linear Function Approximationcited by 15
  9. 2021Compressed Gradient Tracking for Decentralized Optimization Over General Directed Networkscited by 12
  10. 2013Distributed Reinforcement Learning via Gossipcited by 9
  11. 2021Finite-Time Error Bounds for Distributed Linear Stochastic Approximationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Sample and Communication Efficient Fully Decentralized MARL Policy Evaluation via a New Approach: Local TD update
  2. 2025Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective
  3. 2026Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning
  4. 2025A Communication-Efficient Decentralized Actor-Critic Algorithm
  5. 2024One-Shot Averaging for Distributed TD(λ) Under Markov Sampling
  6. 2024Transferable Graphical MARL for Real-Time Estimation in Dynamic Wireless Networks
  7. 2026Adaptive TD-Lambda for Cooperative Multi-agent Reinforcement Learning
  8. 2025Signal attenuation enables scalable decentralized multi-agent reinforcement learning over networks