mlsanityData to 7 Oct 2026

Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage

arXiv 2302.02392 ↗First posted 5 Feb 2023cs.LG stat.MLcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline reinforcement learning (RL) we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, taking the form of assuming some coverage, realizability, Bellman completeness, and/or hard margin (gap). In this work we propose value-based algorithms for offline RL with PAC guarantees under just partial coverage, specifically, coverage of just a single comparator policy, and realizability of soft (entropy-regularized) Q-function of the single policy and a related function defined as a saddle point of certain minimax optimization problem. This offers refined and generally more lax conditions for offline RL. We further show an analogous result for vanilla Q-functions under a soft margin condition. To attain these guarantees, we leverage novel minimax learning algorithms to accurately estimate soft or vanilla Q-functions with L2-convergence guarantees. Our algorithms' loss functions arise from casting the estimation problems as nonlinear convex optimization problems and Lagrangifying.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 54 references that are in the map. First posted on arXiv 5 Feb 2023.

Builds on

42 of 54 references.

Closest ancestors · 23

Same direction, newest first

  1. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning30 Jan
  2. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints28 Dec
  3. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation23 Nov
  4. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian1 Nov
  5. 2022On Gap-dependent Bounds for Offline Reinforcement Learning1 Jun
  6. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps25 Mar
  7. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learning24 Mar
  8. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  9. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  10. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  11. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
  12. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
11 more
  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  2. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  4. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  5. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  6. 2021Fast Rates for the Regret of Offline Reinforcement Learning31 Jan
  7. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  3. 2019A Theory of Regularized Markov Decision Processescited by 279
  4. 2017A unified view of entropy-regularized Markov decision processescited by 174
  5. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  6. 2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 136
  7. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  8. 2019Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPscited by 104
  9. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  10. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  11. 2015External Validity: From Do-Calculus to Transportability Across Populationscited by 75
  12. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
7 more
  1. 2020Fast Rates for Contextual Linear Optimizationcited by 43
  2. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  3. 2020Off-Policy Evaluation and Learning for External Validity under a Covariate Shiftcited by 26
  4. 2017Generalizing causal inferences from individuals in randomized trials to all trial-eligible individualscited by 24
  5. 2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
  6. 2021Gap-Dependent Unsupervised Exploration for Reinforcement Learningcited by 10
  7. 2022Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributionscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 1.

9 papers.

  1. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  2. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Sufficescited by 11
  3. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learningcited by 4
  4. 2024The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximationcited by 4
  5. 2025Inverse Reinforcement Learning with Just Classification and a Few Regressionscited by 4
  6. 2025Towards Optimal Offline Reinforcement Learningcited by 2
  7. 2025Bellman Calibration for V-Learning in Offline Reinforcement Learningcited by 2
  8. 2025Fitted Q-Evaluation without Bellman Completeness via Occupancy Weightingcited by 1
  9. 2026On the Complexity of Offline Reinforcement Learning with Q^star-Approximation and Partial Coveragecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  2. 2023Offline Reinforcement Learning with Additional Covering Distributions
  3. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  4. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
  5. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game
  6. 2023Extreme Q-Learning: MaxEnt RL without Entropy
  7. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble