mlsanityData to 7 Oct 2026

Mildly Constrained Evaluation Policy for Offline Reinforcement Learning

arXiv 2306.03680 ↗First posted 6 Jun 2023cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) methodologies enforce constraints on the policy to adhere closely to the behavior policy, thereby stabilizing value learning and mitigating the selection of out-of-distribution (OOD) actions during test time. Conventional approaches apply identical constraints for both value learning and test time inference. However, our findings indicate that the constraints suitable for value estimation may in fact be excessively restrictive for action selection during test time. To address this issue, we propose a textitMildly Constrained Evaluation Policy (MCEP) for test time inference with a more constrained textittarget policy for value estimation. Since the textittarget policy has been adopted in various prior approaches, MCEP can be seamlessly integrated with them as a plug-in. We instantiate MCEP based on TD3BC (Fujimoto & Gu, 2021), AWAC (Nair et al., 2020) and DQL (Wang et al., 2023) algorithms. The empirical results on D4RL MuJoCo locomotion, high-dimensional humanoid and a set of 16 robotic manipulation tasks show that the MCEP brought significant performance improvement on classic offline RL methods and can further improve SOTA methods. The codes are open-sourced at urlhttps://github.com/egg-west/MCEP.git.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 37 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

32 of 37 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  2. 2022TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets5 Dec
  3. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators29 Nov
  4. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints2 Nov
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  6. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  9. 2021Offline RL Without Off-Policy Evaluation16 Jun
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  12. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
5 more
  1. 2020Critic Regularized Regression26 Jun
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  7. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  8. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  9. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  10. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  11. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  12. 2019Distilling Policy Distillationcited by 81
3 more
  1. 2022Offline Reinforcement Learning for Visual Navigationcited by 28
  2. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  3. 2022Benchmarking Offline Reinforcement Learning Algorithms for E-Commerce Order Fraud Evaluationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Role-Adaptive Policy Optimization for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Policy Constraint by Only Support Constraint for Offline Reinforcement Learning
  2. 2025Test-time Offline Reinforcement Learning on Goal-related Experience
  3. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL
  4. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning
  5. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  6. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
  7. 2021Behavior Constraining in Weight Space for Offline Reinforcement Learning
  8. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning