mlsanityData to 7 Oct 2026

A primal-dual perspective for distributed TD-learning

arXiv 2310.00638 ↗First posted 1 Oct 2023cs.LG math.OCcited by 1

Not on the map. Why?

Abstract (arXiv)

The goal of this paper is to investigate distributed temporal difference (TD) learning for a networked multi-agent Markov decision process. The proposed approach is based on distributed optimization algorithms, which can be interpreted as primal-dual Ordinary differential equation (ODE) dynamics subject to null-space constraints. Based on the exponential convergence behavior of the primal-dual ODE dynamics subject to null-space constraints, we examine the behavior of the final iterate in various distributed TD-learning scenarios, considering both constant and diminishing step-sizes and incorporating both i.i.d. and Markovian observation models. Unlike existing methods, the proposed algorithm does not require the assumption that the underlying communication network structure is characterized by a doubly stochastic matrix.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 65 references that are in the map. First posted on arXiv 1 Oct 2023.

Builds on

24 of 65 references.

References · 24

  1. 2021Convex optimizationcited by 2,479
  2. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  3. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 165
  4. 2014EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimizationcited by 160
  5. 2018Distributed Stochastic Gradient Tracking Methodscited by 106
  6. 2013Distributed optimization over time-varying directed graphscited by 77
  7. 2018Push-Pull Gradient Methods for Distributed Optimization in Networkscited by 45
  8. 2018Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimizationcited by 40
  9. 2013Distributed Policy Evaluation Under Multiple Behavior Strategiescited by 17
  10. 2015DSA: Decentralized Double Stochastic Averaging Gradient Algorithmcited by 14
  11. 2017Exact Diffusion for Distributed Optimization and Learning --- Part II: Convergence Analysiscited by 14
  12. 2019Finite-Time Performance of Distributed Temporal Difference Learning with Linear Function Approximationcited by 14
12 more
  1. 2018Multi-Agent Fully Decentralized Value Function Learning with Linear Convergence Ratescited by 13
  2. 2018On the Exponential Stability of Primal-Dual Gradient Dynamicscited by 12
  3. 2014Distributed convex optimization via continuous-time coordination algorithms with discrete-time communicationcited by 11
  4. 2019A Multistep Lyapunov Approach for Finite-Time Analysis of Biased Stochastic Approximationcited by 10
  5. 2020Finite-Time Convergence Rates of Decentralized Stochastic Approximation with Applications in Multi-Agent and Multi-Task Learningcited by 10
  6. 2013Distributed Reinforcement Learning via Gossipcited by 8
  7. 2018Primal-Dual Algorithm for Distributed Reinforcement Learning: Distributed GTDcited by 8
  8. 2021New Versions of Gradient Temporal Difference Learningcited by 6
  9. 2016Passivity-Based Distributed Optimization with Communication Delays Using PI Consensus Algorithmcited by 3
  10. 2019Fast Multi-Agent Temporal-Difference Learning via Homotopy Stochastic Primal-Dual Optimizationcited by 3
  11. 2022Dual Seminorms, Ergodic Coefficients and Semicontraction Theorycited by 3
  12. 2022Stability, Linear Convergence, and Robustness of the Wang-Elia Algorithm for Distributed Consensus Optimizationcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024A finite time analysis of distributed Q-learningcited by 1