mlsanityData to 7 Oct 2026

Augmenting Offline RL with Unlabeled Data

arXiv 2406.07117 ↗First posted 11 Jun 2024cs.AI cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recent advancements in offline Reinforcement Learning (Offline RL) have led to an increased focus on methods based on conservative policy updates to address the Out-of-Distribution (OOD) issue. These methods typically involve adding behavior regularization or modifying the critic learning objective, focusing primarily on states or actions with substantial dataset support. However, we challenge this prevailing notion by asserting that the absence of an action or state from a dataset does not necessarily imply its suboptimality. In this paper, we propose a novel approach to tackle the OOD problem. We introduce an offline RL teacher-student framework, complemented by a policy similarity measure. This framework enables the student policy to gain insights not only from the offline RL dataset but also from the knowledge transferred by a teacher policy. The teacher policy is trained using another dataset consisting of state-action pairs, which can be viewed as practical domain knowledge acquired without direct interaction with the environment. We believe this additional knowledge is key to effectively solving the OOD issue. This research represents a significant advancement in integrating a teacher-student network into the actor-critic framework, opening new avenues for studies on knowledge transfer in offline RL and effectively addressing the OOD challenge.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 47 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

38 of 47 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption19 Oct
  2. 2023Offline Reinforcement Learning with Imbalanced Datasets6 Jul
  3. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  4. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning27 Feb
  5. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints2 Nov
  6. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  7. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  8. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  9. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
  10. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  11. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
4 more
  1. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019Bidirectional Learning for Domain Adaptation of Semantic Segmentationcited by 308
  8. 2020A Simple Semi-Supervised Learning Framework for Object Detectioncited by 282
  9. 2022Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labelscited by 218
  10. 2017Teacher-Student Curriculum Learningcited by 197
  11. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  12. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
10 more
  1. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  2. 2018Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender Systemcited by 60
  3. 2021Data-Uncertainty Guided Multi-Phase Learning for Semi-Supervised Object Detectioncited by 48
  4. 2023Semi-Supervised Semantic Segmentation via Gentle Teaching Assistantcited by 34
  5. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
  6. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  7. 2019Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learningcited by 12
  8. 2021Off-Policy Correction for Deep Deterministic Policy Gradient Algorithms via Batch Prioritized Experience Replaycited by 3
  9. 2022Safe and Robust Experience Sharing for Deterministic Policy Gradient Algorithmscited by 2
  10. 2022On the Reuse Bias in Off-Policy Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  2. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  3. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  4. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  5. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  6. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
  7. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  8. 2026Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models