# SMIX(λ): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/1911.04094 · HTML: https://mlsanity.com/paper/1911.04094/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2019.

## Built on by (most cited first, 15 of 15)

- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00136/) (2021)
- [A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising](https://mlsanity.com/paper/2106.06224/) (2021)
- [Multi-agent Continual Coordination via Progressive Task Contextualization](https://mlsanity.com/paper/2305.13937/) (2023)
- [Effective Multi-Agent Deep Reinforcement Learning Control with Relative Entropy Regularization](https://mlsanity.com/paper/2309.14727/) (2023)
- [Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2211.12075/) (2022)
- [Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.12957/) (2021)
- [Off-Policy Correction For Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2111.11229/) (2021)
- [Remember and Forget Experience Replay for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2203.13319/) (2022)
- [Group-Agent Reinforcement Learning](https://mlsanity.com/paper/2202.05135/) (2022)
- [Never Explore Repeatedly in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.09909/) (2023)
- [Autonomous Decision Making for UAV Cooperative Pursuit-Evasion Game with Reinforcement Learning](https://mlsanity.com/paper/2411.02983/) (2024)
- [Maximum Correntropy Value Decomposition for Multi-agent Deep Reinforcemen Learning](https://mlsanity.com/paper/2208.03663/) (2022)
- [Multi-Agent Cross-Entropy Method with Monotonic Nonlinear Critic Decomposition](https://mlsanity.com/paper/2511.18671/) (2025)
- [Adaptive TD-Lambda for Cooperative Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2605.11880/) (2026)

## Abstract (arXiv)

Learning a stable and generalizable centralized value function (CVF) is a crucial but challenging task in multi-agent reinforcement learning (MARL), as it has to deal with the issue that the joint action space increases exponentially with the number of agents in such scenarios. This paper proposes an approach, named SMIX(λ), to address the issue using an efficient off-policy centralized training method within a flexible learner search space. As importance sampling for such off-policy training is both computationally costly and numerically unstable, we proposed to use the λ-return as a proxy to compute the TD error. With this new loss function objective, we adopt a modified QMIX network structure as the base to train our model. By further connecting it with the Q(λ) approach from an unified expectation correction viewpoint, we show that the proposed SMIX(λ) is equivalent to Q(λ) and hence shares its convergence properties, while without being suffered from the aforementioned curse of dimensionality problem inherent in MARL. Experiments on the StarCraft Multi-Agent Challenge (SMAC) benchmark demonstrate that our approach not only outperforms several state-of-the-art MARL methods by a large margin, but also can be used as a general tool to improve the overall performance of other CTDE-type algorithms by enhancing their CVFs.