mlsanityData to 7 Oct 2026

The In-Sample Softmax for Offline Reinforcement Learning

arXiv 2302.14372 ↗First posted 28 Feb 2023cs.LG cs.AIcited by 33

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reinforcement learning (RL) agents can leverage batches of previously collected data to extract a reasonable control policy. An emerging issue in this offline RL setting, however, is that the bootstrapping update underlying many of our methods suffers from insufficient action-coverage: standard max operator may select a maximal action that has not been seen in the dataset. Bootstrapping from these inaccurate values can lead to overestimation and even divergence. There are a growing number of methods that attempt to approximate an in-sample max, that only uses actions well-covered by the dataset. We highlight a simple fact: it is more straightforward to approximate an in-sample softmax using only actions in the dataset. We show that policy iteration based on the in-sample softmax converges, and that for decreasing temperatures it approaches the in-sample max. We derive an In-Sample Actor-Critic (AC), using this in-sample softmax, and show that it is consistently better or comparable to existing offline RL methods, and is also well-suited to fine-tuning.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 37 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

27 of 37 references.

Closest ancestors · 17

Same direction, newest first

  1. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning14 Jun
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  3. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  4. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Offline RL Without Off-Policy Evaluation16 Jun
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
5 more
  1. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 10

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  9. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  10. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 11; 2025: 3; 2026: 10.

33 papers.

Sort:

By area:

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  2. 2023Boosting Continuous Control with Consistency Policycited by 36
  3. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  4. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
  5. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guaranteescited by 11
  6. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learningcited by 10
  7. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 10
  8. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
  9. 2024Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximationcited by 10
  10. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
  11. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimizationcited by 7
  12. 2025Mitigating Preference Hacking in Policy Optimization with Pessimismcited by 7
21 more
  1. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexaminationcited by 5
  2. 2023Factual and Personalized Recommendations using Language Models and Reinforcement Learningcited by 4
  3. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency modelcited by 4
  4. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learningcited by 3
  5. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  6. 2026Chunk-Guided Q-Learningcited by 3
  7. 2024q-exponential family for policy optimizationcited by 2
  8. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
  9. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
  10. 2026SPAR: Support-Preserving Action Rectificationcited by 1
  11. 2023Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Explorationcited by 0
  12. 2024Are Expressive Models Truly Necessary for Offline RL?cited by 0
  13. 2025Policy Constraint by Only Support Constraint for Offline Reinforcement Learningcited by 0
  14. 2025Fine-Tuning without Performance Degradationcited by 0
  15. 2026Automatic Constraint Policy Optimization based on Continuous Constraint Interpolation Framework for Offline Reinforcement Learningcited by 0
  16. 2026GAS: Enhancing Reward-Cost Balance of Generative Model-assisted Offline Safe RLcited by 0
  17. 2026kappa-Explorer: A Unified Framework for Active Model Estimation in MDPscited by 0
  18. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  19. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learningcited by 0
  20. 2026Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RLcited by 0
  21. 2026Lifted Bellman Linear Programming for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  2. 2021The Least Restriction for Offline Reinforcement Learning
  3. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  4. 2026Maximum Likelihood Reinforcement Learning
  5. 2023Offline Reinforcement Learning with Additional Covering Distributions
  6. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
  7. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  8. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization