mlsanityData to 14 Oct 2026

Meta-Value Learning: a General Framework for Learning with Learning Awareness

arXiv 2307.08863 ↗First posted 17 Jul 2023cs.LG cs.MAcited by 6

Not on the map. Why?

Abstract (arXiv)

Gradient-based learning in multi-agent systems is difficult because the gradient derives from a first-order model which does not account for the interaction between agents' learning processes. LOLA (arXiv:1709.04326) accounts for this by differentiating through one step of optimization. We propose to judge joint policies by their long-term prospects as measured by the meta-value, a discounted sum over the returns of future optimization iterates. We apply a form of Q-learning to the meta-game of optimization, in a way that avoids the need to explicitly represent the continuous action space of policy updates. The resulting method, MeVa, is consistent and far-sighted, and does not require REINFORCE estimators. We analyze the behavior of our method on a toy game and compare to prior work on repeated matrix games.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

22 of 43 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  8. 2017Noisy Networks for Explorationcited by 266
  9. 2015Highway Networkscited by 260
  10. 2017Learning with Opponent-Learning Awarenesscited by 222
  11. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  12. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
10 more
  1. 2018Inequity aversion improves cooperation in intertemporal social dilemmascited by 106
  2. 2018Stable Opponent Shaping in Differentiable Gamescited by 49
  3. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
  4. 2022Model-Free Opponent Shapingcited by 44
  5. 2020A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learningcited by 41
  6. 2022COLA: Consistent Learning with Opponent-Learning Awarenesscited by 38
  7. 2020Policy Evaluation Networkscited by 28
  8. 2020Emergent Reciprocity and Team Formation from Randomized Uncertain Social Preferencescited by 21
  9. 2022Proximal Learning With Opponent-Learning Awarenesscited by 20
  10. 2022Influencing Long-Term Behavior in Multiagent Reinforcement Learningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 0; 2026: 1.

6 papers.

  1. 2024Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmascited by 12
  2. 2024LOQA: Learning with Opponent Q-Learning Awarenesscited by 8
  3. 2024Advantage Alignment Algorithmscited by 7
  4. 2024BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shapingcited by 4
  5. 2024Multi-agent cooperation through learning-aware policy gradientscited by 4
  6. 2026Multi-agent cooperation through in-context co-player inferencecited by 1