mlsanityData to 7 Oct 2026

DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning

arXiv 2102.07936 ↗First posted 16 Feb 2021cs.MA cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

In fully cooperative multi-agent reinforcement learning (MARL) settings, the environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of the other agents. To address the above issues, we integrate distributional RL and value function factorization methods by proposing a Distributional Value Function Factorization (DFAC) framework to generalize expected value function factorization methods to their DFAC variants. DFAC extends the individual utility functions from deterministic variables to random variables, and models the quantile function of the total return as a quantile mixture. To validate DFAC, we demonstrate DFAC's ability to factorize a simple two-step matrix game with stochastic rewards and perform experiments on all Super Hard tasks of StarCraft Multi-Agent Challenge, showing that DFAC is able to outperform expected value function factorization baselines.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 19 of its 30 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

19 of 30 references.

References · 19

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  3. 2019The StarCraft Multi-Agent Challengecited by 589
  4. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  6. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  7. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  8. 2020ROMA: Multi-Agent Reinforcement Learning with Emergent Rolescited by 135
  9. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  10. 2019Learning Nearly Decomposable Value Functions Via Communication Minimizationcited by 85
  11. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  12. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 59
7 more
  1. 2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
  2. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  3. 2019Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Controlcited by 46
  4. 2018Multi-Agent Common Knowledge Reinforcement Learningcited by 31
  5. 2019Distributional reinforcement learning with linear function approximationcited by 15
  6. 2018QUOTA: The Quantile Option Architecture for Reinforcement Learningcited by 13
  7. 2019Distributional Reward Decomposition for Reinforcement Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 6; 2024: 4; 2025: 3; 2026: 3.

26 papers.

Sort:

By area:

  1. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  2. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  3. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
  4. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  5. 2024Foundations of Multivariate Distributional Reinforcement Learningcited by 16
  6. 2021Divergence-Regularized Multi-Agent Actor-Criticcited by 15
  7. 2022Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learningcited by 6
  8. 2022Risk Perspective Exploration in Distributional Reinforcement Learningcited by 5
  9. 2024Inferring Latent Temporal Sparse Coordination Graph for Multi-Agent Reinforcement Learningcited by 5
  10. 2023Noise Distribution Decomposition based Multi-Agent Distributional Reinforcement Learningcited by 4
  11. 2022MCMARL: Parameterizing Value Function via Mixture of Categorical Distributions for Multi-Agent Reinforcement Learningcited by 3
  12. 2024QTypeMix: Enhancing Multi-Agent Cooperative Strategies through Heterogeneous and Homogeneous Value Decompositioncited by 3
14 more
  1. 2022Decision-making with Speculative Opponent Modelscited by 2
  2. 2022The StarCraft Multi-Agent Challenges+ : Learning of Multi-Stage Tasks and Environmental Factors without Precise Reward Functionscited by 1
  3. 2023Dual Self-Awareness Value Decomposition Framework without Individual Global Max for Cooperative Multi-Agent Reinforcement Learningcited by 1
  4. 2023Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learningcited by 1
  5. 2023A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learningcited by 1
  6. 2025Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamicscited by 1
  7. 2022Exploiting Semantic Epsilon Greedy Exploration Strategy in Multi-Agent Reinforcement Learningcited by 0
  8. 2022Efficient CDF Approximations for Normalizing Flowscited by 0
  9. 2024Collaborative Reinforcement Learning Based Unmanned Aerial Vehicle (UAV) Trajectory Design for 3D UAV Trackingcited by 0
  10. 2025Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learningcited by 0
  11. 2025Multi-Agent Cross-Entropy Method with Monotonic Nonlinear Critic Decompositioncited by 0
  12. 2026Episodic Memory Temporal Consistency for Cooperative Multi-Agent Reinforcement Learningcited by 0
  13. 2026Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methodscited by 0
  14. 2026Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMscited by 0