mlsanityData to 14 Oct 2026

Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset

arXiv 2502.18955 ↗First posted 26 Feb 2025cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) represents a significant shift in RL research, allowing agents to learn from pre-collected datasets without further interaction with the environment. A key, yet underexplored, challenge in offline RL is selecting an optimal subset of the offline dataset that enhances both algorithm performance and training efficiency. Reducing dataset size can also reveal the minimal data requirements necessary for solving similar problems. In response to this challenge, we introduce ReDOR (Reduced Datasets for Offline RL), a method that frames dataset selection as a gradient approximation optimization problem. We demonstrate that the widely used actor-critic framework in RL can be reformulated as a submodular optimization objective, enabling efficient subset selection. To achieve this, we adapt orthogonal matching pursuit (OMP), incorporating several novel modifications tailored for offline RL. Our experimental results show that the data subsets identified by ReDOR not only boost algorithm performance but also do so with significantly lower computational complexity.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 54 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

42 of 54 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets6 Oct
  2. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting22 Jun
  3. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discovery2 Dec
  4. 2022Boosting Offline Reinforcement Learning via Data Rebalancing17 Oct
  5. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  6. 2022On the Role of Discount Factor in Offline Reinforcement Learning7 Jun
  7. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning31 Jan
  8. 2021Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning26 Nov
  9. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  10. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  11. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
7 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020MOPO: Model-based Offline Policy Optimization27 May
  4. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 23

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2015Prioritized Experience Replaycited by 1,099
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  6. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  10. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  11. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  12. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 226
11 more
  1. 2020The Cost of Training NLP Models: A Concise Overviewcited by 98
  2. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  3. 2020Submodular Combinatorial Information Measures with Applications in Machine Learningcited by 88
  4. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  5. 2021RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learningcited by 74
  6. 2019Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Visioncited by 64
  7. 2016Restricted Strong Convexity Implies Weak Submodularitycited by 52
  8. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 17
  9. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applicationscited by 17
  10. 2020Improved Guarantees for k-means++ and k-means++ Parallelcited by 12
  11. 2022Offline Reinforcement Learning for Mobile Notificationscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025ICR-RL: Deep Reinforcement Learning via In-Context Regressioncited by 8
  2. 2026Position: Stop Preaching and Start Practising Data Frugality for Responsible Development of AIcited by 0
  3. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Explorationcited by 0
  4. 2026CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity
  2. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  3. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  4. 2021The Least Restriction for Offline Reinforcement Learning
  5. 2020Critic Regularized Regression
  6. 2022Leveraging Offline Data in Online Reinforcement Learning
  7. 2020Hyperparameter Selection for Offline Reinforcement Learning
  8. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data