# Multi-Agent Fully Decentralized Value Function Learning with Linear Convergence Rates

arXiv: https://arxiv.org/abs/1810.07792 · HTML: https://mlsanity.com/paper/1810.07792/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2018.

## Built on by (most cited first, 13 of 13)

- [Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysis](https://mlsanity.com/paper/2109.03699/) (2021)
- [Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.06042/) (2021)
- [Multi-Agent Off-Policy TD Learning: Finite-Time Analysis with Near-Optimal Sample Complexity and Communication Complexity](https://mlsanity.com/paper/2103.13147/) (2021)
- [Who Needs to Know? Minimal Knowledge for Optimal Coordination](https://mlsanity.com/paper/2306.09309/) (2023)
- [Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph](https://mlsanity.com/paper/2201.04962/) (2022)
- [TAG: A Decentralized Framework for Multi-Agent Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2502.15425/) (2025)
- [Distributed Policy Gradient with Variance Reduction in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2111.12961/) (2021)
- [Federated Reinforcement Learning at the Edge](https://mlsanity.com/paper/2112.05908/) (2021)
- [Policy Evaluation in Decentralized POMDPs with Belief Sharing](https://mlsanity.com/paper/2302.04151/) (2023)
- [A primal-dual perspective for distributed TD-learning](https://mlsanity.com/paper/2310.00638/) (2023)
- [An Algorithm For Adversary Aware Decentralized Networked MARL](https://mlsanity.com/paper/2305.05573/) (2023)
- [Continuous-Time Distributed Dynamic Programming for Networked Multi-Agent Markov Decision Processes](https://mlsanity.com/paper/2307.16706/) (2023)
- [Reinforcement Networks: novel framework for collaborative Multi-Agent Reinforcement Learning tasks](https://mlsanity.com/paper/2512.22876/) (2025)

## Abstract (arXiv)

This work develops a fully decentralized multi-agent algorithm for policy evaluation. The proposed scheme can be applied to two distinct scenarios. In the first scenario, a collection of agents have distinct datasets gathered following different behavior policies (none of which is required to explore the full state space) in different instances of the same environment and they all collaborate to evaluate a common target policy. The network approach allows for efficient exploration of the state space and allows all agents to converge to the optimal solution even in situations where neither agent can converge on its own without cooperation. The second scenario is that of multi-agent games, in which the state is global and rewards are local. In this scenario, agents collaborate to estimate the value function of a target team policy. The proposed algorithm combines off-policy learning, eligibility traces and linear function approximation. The proposed algorithm is of the variance-reduced kind and achieves linear convergence with O(1) memory requirements. The linear convergence of the algorithm is established analytically, and simulations are used to illustrate the effectiveness of the method.