mlsanityData to 14 Oct 2026

Beyond Pessimism: Offline Learning in KL-regularized Games

arXiv 2604.06738 ↗First posted 8 Apr 2026cs.GT cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We study offline learning in KL-regularized two-player zero-sum games, where policies are optimized with respect to a fixed reference policy through KL regularization. Prior work relies on pessimistic value estimation to handle distribution shift, yielding only widetildeO(1/√ n) statistical rates. We develop a new pessimism-free algorithm and analytical framework for KL-regularized games, built on the smoothness of KL-regularized best responses and a stability property of the Nash equilibrium induced by skew symmetry. This yields, to our knowledge, the first pessimism-free offline learning guarantee for KL-regularized games, with a fast widetildeO(1/n) sample complexity bound. We further propose an efficient self-play policy optimization algorithm that replaces exact equilibrium computation with iterative KL-regularized policy updates, and prove that its last iterate preserves the same pessimism-free statistical guarantee up to a controlled optimization error.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 8 Apr 2026.

Builds on

24 of 36 references.

References · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  7. 2023Nash Learning from Human Feedbackcited by 224
  8. 2013Optimization, Learning, and Games with Predictable Sequencescited by 188
  9. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  10. 2015Fast Convergence of Regularized Learning in Gamescited by 150
  11. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  12. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
12 more
  1. 2020Batch Value-function Approximation with Only Realizabilitycited by 104
  2. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 83
  3. 2020Hedging in games: Faster convergence of external and swap regretscited by 57
  4. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  5. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 44
  6. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 30
  7. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 29
  8. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  9. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 22
  10. 2023Offline Learning in Markov Games with General Function Approximationcited by 18
  11. 2025Achieving Logarithmic Regret in KL-Regularized Zero-Sum Markov Gamescited by 15
  12. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Offline Two-Player Zero-Sum Markov Games with KL Regularizationcited by 6
  2. 2026Pessimism-Free Offline Learning in General-Sum Games via KL Regularizationcited by 5
  3. 2026On the Optimal Sample Complexity of Offline Multi-Armed Bandits with KL Regularizationcited by 4
  4. 2026Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrabilitycited by 3
  5. 2026Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluationcited by 2
  6. 2026OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Itemscited by 1
  7. 2026Minimax Last-Iterate Convergence in Matrix Games with Observed Actionscited by 1
  8. 2026Context-dependent agent evaluation with orthogonal equilibrium learningcited by 0
  9. 2026Pessimistic Minimax Learning for Public-Private Information Games under Unilateral Coveragecited by 0
  10. 2026Fast Rates for Offline Q-Learning with KL Regularizationcited by 0