mlsanityData to 14 Oct 2026

Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification

arXiv 2606.06053 ↗First posted 4 Jun 2026cs.LGcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We study KL-regularized contextual bandits and episodic reinforcement learning (RL) under general function approximation with model misspecification. Existing guarantees rely on realizability and therefore do not extend to misspecified models, where classical regret bounds may fail. This work introduces KL misspecification formulations for contextual bandits and episodic RL and analyzes regression-based algorithms with Gibbs policy updates. High-probability KL-regret guarantees with explicit misspecification terms are established, recovering the standard realizable KL-regularized setting as a special case.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 33 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

28 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learning11 Feb
  2. 2024Mitigating Covariate Shift in Misspecified Regression with Applications to Reinforcement Learning22 Jan

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  6. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  7. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 201
  8. 2020Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oraclescited by 197
  9. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  10. 2023Is RLHF More Difficult than Standard RL?cited by 96
  11. 2018Practical Contextual Bandits with Regression Oraclescited by 93
  12. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
14 more
  1. 2021Adapting to Misspecification in Contextual Banditscited by 81
  2. 2020Instance-Dependent Complexity of Contextual Bandits and Reinforcement Learning: A Disagreement-Based Perspectivecited by 78
  3. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  4. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  5. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  6. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
  7. 2021Adapting to Misspecification in Contextual Bandits with Offline Regression Oraclescited by 21
  8. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 11
  9. 2023On the Model-Misspecification in Reinforcement Learningcited by 7
  10. 2021Improved Algorithms for Misspecified Linear Markov Decision Processescited by 6
  11. 2025Regret Bounds for Adversarial Contextual Bandits with General Function Approximation and Delayed Feedbackcited by 5
  12. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  13. 2026Taming the Monster Every Context: Complexity Measure and Unified Framework for Offline-Oracle Efficient Contextual Banditscited by 4
  14. 2026Optimal Regret for Policy Optimization in Contextual Banditscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026When Greedy Sampling Explores: KL-Regularized Contextual Bandits without Eluder-Dimension Dependencecited by 1
  2. 2026Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectivescited by 0
  3. 2026Fast Rates for Offline Q-Learning with KL Regularizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Near-Optimal Regret for KL-Regularized Multi-Armed Bandits
  2. 2026Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability
  3. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
  4. 2025Achieving Logarithmic Regret in KL-Regularized Zero-Sum Markov Games
  5. 2024Kernel-Based Function Approximation for Average Reward Reinforcement Learning: An Optimist No-Regret Algorithm
  6. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapse
  7. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Bandits
  8. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization