mlsanityData to 14 Oct 2026

Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysis

arXiv 2503.17454 ↗First posted 21 Mar 2025cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

Federated reinforcement learning (FedRL) enables collaborative learning while preserving data privacy by preventing direct data exchange between agents. However, many existing FedRL algorithms assume that all agents operate in identical environments, which is often unrealistic. In real-world applications, such as multi-robot teams, crowdsourced systems, and large-scale sensor networks, each agent may experience slightly different transition dynamics, leading to inherent model mismatches. In this paper, we first establish linear convergence guarantees for single-agent temporal difference learning (TD(0)) in policy evaluation and demonstrate that under a perturbed environment, the agent suffers a systematic bias that prevents accurate estimation of the true value function. This result holds under both i.i.d. and Markovian sampling regimes. We then extend our analysis to the federated TD(0) (FedTD(0)) setting, where multiple agents, each interacting with its own perturbed environment, periodically share value estimates to collaboratively approximate the true value function of a common underlying model. Our theoretical results indicate the impact of model mismatch, network connectivity, and mixing behavior on the convergence of FedTD(0). Empirical experiments corroborate our theoretical gains, highlighting that even moderate levels of information sharing significantly mitigate environment-specific errors.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 56 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

31 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Compressed Federated Reinforcement Learning with a Generative Model26 Mar
  2. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning27 Jan

Other references · 29

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  3. 2017Robust Adversarial Reinforcement Learningcited by 451
  4. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
  5. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  6. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 166
  7. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  8. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 78
  9. 2017Sub-sampled Cubic Regularization for Non-convex Optimizationcited by 63
  10. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  11. 2021A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variantscited by 57
  12. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
17 more
  1. 2022Adapting to Mixing Time in Stochastic Optimization with Markovian Datacited by 36
  2. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 36
  3. 2019Federated Deep Reinforcement Learningcited by 35
  4. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  5. 2021Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysiscited by 26
  6. 2020Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedupcited by 25
  7. 2023Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stabilitycited by 20
  8. 2023Federated Q-Learning: Linear Regret Speedup with Low Communication Costcited by 19
  9. 2019Characterizing the Exact Behaviors of Temporal Difference Learning Algorithms Using Markov Jump Linear System Theorycited by 17
  10. 2019Gossip-based Actor-Learner Architectures for Deep Reinforcement Learningcited by 16
  11. 2024A Simple Finite-Time Analysis of TD Learning with Linear Function Approximationcited by 16
  12. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 15
  13. 2023Distributed TD(0) with Almost No Communicationcited by 13
  14. 2018Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximationcited by 12
  15. 2023Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learningcited by 12
  16. 2020On the Convergence of Consensus Algorithms with Markovian Noise and Gradient Biascited by 10
  17. 2023Comparing NARS and Reinforcement Learning: An Analysis of ONA and Q-Learning Algorithmscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agentscited by 2
  2. 2026Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradientscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning
  2. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments
  3. 2026Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Cost
  4. 2026Personalized Observation Normalization for Federated Reinforcement Learning in Simulation Environments with Heterogeneity
  5. 2024On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations
  6. 2025Federated Reinforcement Learning in Heterogeneous Environments
  7. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
  8. 2026FedQHD: Closed-Form Function-Space Federated Reinforcement Learning