mlsanityData to 7 Oct 2026

Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation

arXiv 2502.00601 ↗First posted 2 Feb 2025cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The success of deep reinforcement learning (DRL) relies on the availability and quality of training data, often requiring extensive interactions with specific environments. In many real-world scenarios, where data collection is costly and risky, offline reinforcement learning (RL) offers a solution by utilizing data collected by domain experts and searching for a batch-constrained optimal policy. This approach is further augmented by incorporating external data sources, expanding the range and diversity of data collection possibilities. However, existing offline RL methods often struggle with challenges posed by non-matching data from these external sources. In this work, we specifically address the problem of source-target domain mismatch in scenarios involving mixed datasets, characterized by a predominance of source data generated from random or suboptimal policies and a limited amount of target data generated from higher-quality policies. To tackle this problem, we introduce Transition Scoring (TS), a novel method that assigns scores to transitions based on their similarity to the target domain, and propose Curriculum Learning-Based Trajectory Valuation (CLTV), which effectively leverages these transition scores to identify and prioritize high-quality trajectories through a curriculum learning approach. Our extensive experiments across various offline RL methods and MuJoCo environments, complemented by rigorous theoretical analysis, demonstrate that CLTV enhances the overall performance and transferability of policies learned by offline RL algorithms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 44 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

25 of 44 references.

Closest ancestors · 7

Same direction, newest first

  1. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 18

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2015Prioritized Experience Replaycited by 1,070
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  9. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  10. 2018Distributed Prioritized Experience Replaycited by 260
  11. 2019Data Valuation using Reinforcement Learningcited by 115
  12. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
6 more
  1. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
  2. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  3. 2019ChainerRL: A Deep Reinforcement Learning Librarycited by 51
  4. 2019Safe Reinforcement Learning on Autonomous Vehiclescited by 48
  5. 2018Macro action selection with deep reinforcement learning in StarCraftcited by 6
  6. 2022SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisitioncited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Data Valuation for Offline Reinforcement Learning
  2. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  3. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  4. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  5. 2025Expert or not? assessing data quality in offline reinforcement learning
  6. 2026Not All Rollouts Are Worth Learning: On Trajectory Valuation for Post-Training Reinforcement Learning
  7. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning
  8. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correction