mlsanityData to 7 Oct 2026

Dataset Clustering for Improved Offline Policy Learning

arXiv 2402.09550 ↗First posted 14 Feb 2024cs.LG cs.ROcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline policy learning aims to discover decision-making policies from previously-collected datasets without additional online interactions with the environment. As the training dataset is fixed, its quality becomes a crucial determining factor in the performance of the learned policy. This paper studies a dataset characteristic that we refer to as multi-behavior, indicating that the dataset is collected using multiple policies that exhibit distinct behaviors. In contrast, a uni-behavior dataset would be collected solely using one policy. We observed that policies learned from a uni-behavior dataset typically outperform those learned from multi-behavior datasets, despite the uni-behavior dataset having fewer examples and less diversity. Therefore, we propose a behavior-aware deep clustering approach that partitions multi-behavior datasets into several uni-behavior subsets, thereby benefiting downstream policy learning. Our approach is flexible and effective; it can adaptively estimate the number of clusters while demonstrating high clustering accuracy, achieving an average Adjusted Rand Index of 0.987 across various continuous control task datasets. Finally, we present improved policy learning examples using dataset clustering and discuss several potential scenarios where our approach might benefit the offline policy learning community.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 59 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

36 of 59 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning23 Jun
  2. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size20 Nov
  3. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Reducing Conservativeness Oriented Offline Reinforcement Learning27 Feb
  8. 2020Critic Regularized Regression26 Jun
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016OpenAI Gymcited by 1,684
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  9. 2016Deep Exploration via Bootstrapped DQNcited by 518
  10. 2018Learning from positive and unlabeled data: a surveycited by 218
  11. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  12. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
14 more
  1. 2022A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directionscited by 73
  2. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  3. 2023Single-model uncertainty quantification in neural network potentials does not consistently outperform model ensemblescited by 30
  4. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  5. 2023Ensemble Reinforcement Learning: A Surveycited by 16
  6. 2020SEERL: Sample Efficient Ensemble Reinforcement Learningcited by 11
  7. 2023Improving Behavioural Cloning with Positive Unlabeled Learningcited by 8
  8. 2020Reinforcement Learning based dynamic weighing of Ensemble Models for Time Series Forecastingcited by 6
  9. 2019Clustered Reinforcement Learningcited by 5
  10. 2021Solving the Real Robot Challenge using Deep Reinforcement Learningcited by 5
  11. 2022Dexterous Robotic Manipulation using Deep Reinforcement Learning and Knowledge Transfer for Complex Sparse Reward-based Taskscited by 5
  12. 2023Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policiescited by 5
  13. 2022Out-of-Dynamics Imitation Learning from Multimodal Demonstrationscited by 3
  14. 2023Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real Worldcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Less is More: Clustered Cross-Covariance Control for Offline RLcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Policy-Based Trajectory Clustering in Offline Reinforcement Learning
  2. 2025Offline Behavioral Data Selection
  3. 2025State Diversity Matters in Offline Behavior Distillation
  4. 2022Behavior Estimation from Multi-Source Data for Offline Reinforcement Learning
  5. 2021A Dataset Perspective on Offline Reinforcement Learning
  6. 2026JumpStart Your Policy Learning with Lessons from 160,000 Training Runs
  7. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  8. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning