mlsanityData to 14 Oct 2026

CUDC: A Curiosity-Driven Unsupervised Data Collection Method with Adaptive Temporal Distances for Offline Reinforcement Learning

arXiv 2312.12191 ↗First posted 19 Dec 2023cs.LG cs.AI stat.MLcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) aims to learn an effective policy from a pre-collected dataset. Most existing works are to develop sophisticated learning algorithms, with less emphasis on improving the data collection process. Moreover, it is even challenging to extend the single-task setting and collect a task-agnostic dataset that allows an agent to perform multiple downstream tasks. In this paper, we propose a Curiosity-driven Unsupervised Data Collection (CUDC) method to expand feature space using adaptive temporal distances for task-agnostic data collection and ultimately improve learning efficiency and capabilities for multi-task offline RL. To achieve this, CUDC estimates the probability of the k-step future states being reachable from the current states, and adapts how many steps into the future that the dynamics model should predict. With this adaptive reachability mechanism in place, the feature representation can be diversified, and the agent can navigate itself to collect higher-quality data with curiosity. Empirically, CUDC surpasses existing unsupervised methods in efficiency and learning performance in various downstream offline RL tasks of the DeepMind control suite.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 37 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

25 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 22

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2016Layer Normalizationcited by 5,669
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  8. 2018DeepMind Control Suitecited by 956
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  10. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  11. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  12. 2019Self-Supervised Exploration via Disagreementcited by 265
10 more
  1. 2019Efficient Exploration via State Marginal Matchingcited by 188
  2. 2021Reinforcement Learning with Prototypical Representationscited by 179
  3. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  4. 2021APS: Active Pretraining with Successor Featurescited by 112
  5. 2018Episodic Curiosity through Reachabilitycited by 109
  6. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  7. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  8. 2021A Data-Centric Approach for Training Deep Neural Networks with Less Datacited by 28
  9. 2018BaRC: Backward Reachability Curriculum for Robotic Reinforcement Learningcited by 23
  10. 2022From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Taskscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022The Challenges of Exploration for Offline Reinforcement Learning
  2. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  3. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
  4. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  5. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  6. 2024State-Constrained Offline Reinforcement Learning
  7. 2023Using Curiosity for an Even Representation of Tasks in Continual Offline Reinforcement Learning
  8. 2021A Minimalist Approach to Offline Reinforcement Learning