# Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

arXiv: https://arxiv.org/abs/2112.15156 · HTML: https://mlsanity.com/paper/2112.15156/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 63 references that are in the map.
- First posted on arXiv 30 Dec 2021.

## Builds on (most cited first, 23 of 23)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks](https://mlsanity.com/paper/1812.09755/) (2018)
- [A Sharp Analysis of Model-based Reinforcement Learning with Self-Play](https://mlsanity.com/paper/2010.01604/) (2020)
- [Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices](https://mlsanity.com/paper/2008.02790/) (2020)
- [Tactical Optimism and Pessimism for Deep Reinforcement Learning](https://mlsanity.com/paper/2102.03765/) (2021)
- [Temporal Abstraction in Reinforcement Learning with the Successor Representation](https://mlsanity.com/paper/2110.05740/) (2021)
- [Measuring the Reliability of Reinforcement Learning Algorithms](https://mlsanity.com/paper/1912.05663/) (2019)
- [Robust Model-free Reinforcement Learning with Multi-objective Bayesian Optimization](https://mlsanity.com/paper/1910.13399/) (2019)
- [PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Exploration](https://mlsanity.com/paper/2107.07410/) (2021)
- [A neurally plausible model learns successor representations in partially observable environments](https://mlsanity.com/paper/1906.09480/) (2019)
- [A Complementary Learning Systems Approach to Temporal Difference Learning](https://mlsanity.com/paper/1905.02636/) (2019)
- [Kalman Temporal Differences](https://mlsanity.com/paper/1406.3270/) (2014)
- [Universal Successor Representations for Transfer Reinforcement Learning](https://mlsanity.com/paper/1804.03758/) (2018)
- [MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learning](https://mlsanity.com/paper/2006.00195/) (2020)
- [Agent-Centric Representations for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2104.09402/) (2021)
- [Probabilistic Successor Representations with Kalman Temporal Differences](https://mlsanity.com/paper/1910.02532/) (2019)
- [Model-Free Optimal Control of Linear Multi-Agent Systems via Decomposition and Hierarchical Approximation](https://mlsanity.com/paper/2008.06604/) (2020)

## Built on by (most cited first, 1 of 1)

- [Hybrid Indoor Localization via Reinforcement Learning-based Information Fusion](https://mlsanity.com/paper/2210.15132/) (2022)

## Abstract (arXiv)

Distributed Multi-Agent Reinforcement Learning (MARL) algorithms has attracted a surge of interest lately mainly due to the recent advancements of Deep Neural Networks (DNNs). Conventional Model-Based (MB) or Model-Free (MF) RL algorithms are not directly applicable to the MARL problems due to utilization of a fixed reward model for learning the underlying value function. While DNN-based solutions perform utterly well when a single agent is involved, such methods fail to fully generalize to the complexities of MARL problems. In other words, although recently developed approaches based on DNNs for multi-agent environments have achieved superior performance, they are still prone to overfiting, high sensitivity to parameter selection, and sample inefficiency. The paper proposes the Multi-Agent Adaptive Kalman Temporal Difference (MAK-TD) framework and its Successor Representation-based variant, referred to as the MAK-SR. Intuitively speaking, the main objective is to capitalize on unique characteristics of Kalman Filtering (KF) such as uncertainty modeling and online second order learning. The proposed MAK-TD/SR frameworks consider the continuous nature of the action-space that is associated with high dimensional multi-agent environments and exploit Kalman Temporal Difference (KTD) to address the parameter uncertainty. By leveraging the KTD framework, SR learning procedure is modeled into a filtering problem, where Radial Basis Function (RBF) estimators are used to encode the continuous space into feature vectors. On the other hand, for learning localized reward functions, we resort to Multiple Model Adaptive Estimation (MMAE), to deal with the lack of prior knowledge on the observation noise covariance and observation mapping function. The proposed MAK-TD/SR frameworks are evaluated via several experiments, which are implemented through the OpenAI Gym MARL benchmarks.