mlsanityData to 7 Oct 2026

NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios

arXiv 2503.19267 ↗First posted 25 Mar 2025cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) aims to learn from historical data without requiring (costly) access to the environment. To facilitate offline RL research, we previously introduced NeoRL, which highlighted that datasets from real-world tasks are often conservative and limited. With years of experience applying offline RL to various domains, we have identified additional real-world challenges. These include extremely conservative data distributions produced by deployed control systems, delayed action effects caused by high-latency transitions, external factors arising from the uncontrollable variance of transitions, and global safety constraints that are difficult to evaluate during the decision-making process. These challenges are underrepresented in previous benchmarks but frequently occur in real-world tasks. To address this, we constructed the extended Near Real-World Offline RL Benchmark (NeoRL-2), which consists of 7 datasets from 7 simulated tasks along with their corresponding evaluation simulators. Benchmarking results from state-of-the-art offline RL approaches demonstrate that current methods often struggle to outperform the data-collection behavior policy, highlighting the need for more effective methods. We hope NeoRL-2 will accelerate the development of reinforcement learning algorithms for real-world applications. The benchmark project page is available at https://github.com/polixir/NeoRL2.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 52 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

29 of 52 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning9 Oct
  2. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  3. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  5. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  8. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020MOPO: Model-based Offline Policy Optimization27 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  5. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  6. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  7. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  8. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  9. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  10. 2022Partially Observable Markov Decision Processes in Robotics: A Surveycited by 81
  11. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  12. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
5 more
  1. 2022A Survey on Model-based Reinforcement Learningcited by 65
  2. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  3. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  4. 2021Generalizing Off-Policy Learning under Sample Selection Biascited by 16
  5. 2022Adversarial Counterfactual Environment Model Learningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regressioncited by 0
  2. 2026Pessimistic Auxiliary Policy for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
  2. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning
  3. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  4. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators
  5. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  6. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library
  7. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
  8. 2020Offline Reinforcement Learning Hands-On