mlsanityData to 7 Oct 2026

Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning

arXiv 2602.17062 ↗First posted 19 Feb 2026cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on a single optimal action and struggle to adapt when the underlying value function shifts during training, often converging to suboptimal policies. To address this limitation, we propose Successive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables Qtot to adjust quickly to the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms, demonstrating improved adaptability and overall performance. Our code is available at https://github.com/hyeon1996/S2Q.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 50 of its 69 references that are in the map. First posted on arXiv 19 Feb 2026.

Builds on

50 of 69 references.

References · 50

  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  2. 2018Exploration by Random Network Distillationcited by 966
  3. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  4. 2019The StarCraft Multi-Agent Challengecited by 589
  5. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  6. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  7. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  8. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  9. 2016Learning Multiagent Communication with Backpropagationcited by 355
  10. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  11. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  12. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
38 more
  1. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  2. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  3. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  4. 2020Weighted QMIX: Expanding Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 197
  5. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  6. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  7. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  8. 2020ROMA: Multi-Agent Reinforcement Learning with Emergent Rolescited by 135
  9. 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
  10. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  11. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  12. 2019Learning Nearly Decomposable Value Functions Via Communication Minimizationcited by 85
  13. 2019Multi-Agent Game Abstraction via Graph Attention Neural Networkcited by 81
  14. 2020Qatten: A General Framework for Cooperative Multiagent Reinforcement Learningcited by 81
  15. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  16. 2020Value-Decomposition Multi-Agent Actor-Criticscited by 47
  17. 2024Learning Multi-Agent Communication from Graph Modeling Perspectivecited by 47
  18. 2021Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Surveycited by 34
  19. 2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
  20. 2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
  21. 2020Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Explorationcited by 20
  22. 2022Contrastive Identity-Aware Learning for Multi-Agent Value Decompositioncited by 19
  23. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  24. 2022ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependencycited by 15
  25. 2023Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackerscited by 15
  26. 2020Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learningcited by 14
  27. 2023NA2Q: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learningcited by 14
  28. 2023Context-aware Communication for Multi-agent Reinforcement Learningcited by 13
  29. 2022Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradientscited by 12
  30. 2023Effective and Stable Role-Based Multi-Agent Collaboration by Structural Information Principlescited by 12
  31. 2023FoX: Formation-aware exploration in multi-agent reinforcement learningcited by 9
  32. 2024T2MAC: Targeted and Trusted Multi-Agent Communication through Selective Engagement and Evidence-Driven Integrationcited by 8
  33. 2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
  34. 2024SMAC-Hard: Enabling Mixed Opponent Strategy Script and Self-play on SMACcited by 5
  35. 2025Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learningcited by 5
  36. 2022Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learningcited by 4
  37. 2023ConcaveQ: Non-Monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learningcited by 3
  38. 2024RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Spacecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learningcited by 2
  2. 2026Interaction-Breaking Adversarial Learning Framework for Robust Multi-Agent Reinforcement Learningcited by 1
  3. 2026Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methodscited by 0