mlsanityData to 7 Oct 2026

Pessimism-Free Offline Learning in General-Sum Games via KL Regularization

arXiv 2605.00264 ↗First posted 30 Apr 2026cs.LG cs.GTcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline multi-agent reinforcement learning in general-sum settings is challenged by the distribution shift between logged datasets and target equilibrium policies. While standard methods rely on manual pessimistic penalties, we demonstrate that KL regularization suffices to stabilize learning and achieve equilibrium recovery. We propose General-sum Anchored Nash Equilibrium (GANE), which recovers regularized Nash equilibria at an accelerated statistical rate of widetildeO(1/n). For computational tractability, we develop General-sum Anchored Mirror Descent (GAMD), an iterative algorithm converging to a Coarse Correlated Equilibrium at the standard rate of widetildeO(1/√n+1/T). These results establish KL regularization as a standalone mechanism for pessimism-free offline learning that achieves equivalent or accelerated rates in multi-player general-sum games.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 39 references that are in the map. First posted on arXiv 30 Apr 2026.

Builds on

29 of 39 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Offline Learning in Markov Games with General Function Approximation6 Feb
  2. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonus1 Jun
  3. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  4. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  5. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  6. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  9. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  10. 2020Batch Value-function Approximation with Only Realizability11 Aug
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  5. 2023Nash Learning from Human Feedbackcited by 220
  6. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  7. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  8. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  9. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  10. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  11. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
  12. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 12
6 more
  1. 2022Model-Based Reinforcement Learning for Offline Zero-Sum Markov Gamescited by 11
  2. 2025Achieving Logarithmic Regret in KL-Regularized Zero-Sum Markov Gamescited by 11
  3. 2024Efficient Multi-Policy Evaluation for Reinforcement Learningcited by 8
  4. 2024Doubly Optimal Policy Evaluation for Reinforcement Learningcited by 7
  5. 2026Beyond Pessimism: Offline Learning in KL-regularized Gamescited by 7
  6. 2024Efficient Policy Evaluation with Safety Constraint for Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Offline Two-Player Zero-Sum Markov Games with KL Regularizationcited by 3
  2. 2026Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluationcited by 1
  3. 2026OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Itemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Scalable Offline Reinforcement Learning for Mean Field Games
  2. 2022Offline Equilibrium Finding
  3. 2026Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning
  4. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?
  5. 2024Offline Fictitious Self-Play for Competitive Games
  6. 2024Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rank
  7. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
  8. 2026NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games