mlsanityData to 14 Oct 2026

Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories

arXiv 2210.06518 ↗First posted 12 Oct 2022cs.LG cs.AI cs.ROcited by 28

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Natural agents can effectively learn from multiple data sources that differ in size, quality, and types of measurements. We study this heterogeneity in the context of offline reinforcement learning (RL) by introducing a new, practically motivated semi-supervised setting. Here, an agent has access to two sets of trajectories: labelled trajectories containing state, action and reward triplets at every timestep, along with unlabelled trajectories that contain only state and reward information. For this setting, we develop and study a simple meta-algorithmic pipeline that learns an inverse dynamics model on the labelled data to obtain proxy-labels for the unlabelled data, followed by the use of any offline RL algorithm on the true and proxy-labelled trajectories. Empirically, we find this simple pipeline to be highly successful -- on several D4RL benchmarks~citefu2020d4rl, certain offline RL algorithms can match the performance of variants trained on a fully labelled dataset even when we label only 10% of trajectories which are highly suboptimal. To strengthen our understanding, we perform a large-scale controlled empirical study investigating the interplay of data-centric properties of the labelled and unlabelled datasets, with algorithmic design choices (e.g., choice of inverse dynamics, offline RL algorithm) to identify general trends and best practices for training RL agents on semi-supervised offline datasets.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 59 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

44 of 59 references.

Closest ancestors · 14

Same direction, newest first

  1. 2022Offline Reinforcement Learning at Multiple Frequencies26 Jul
  2. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  3. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
  4. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  5. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions29 Nov
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  9. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  10. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
2 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2016Generative Adversarial Imitation Learningcited by 1,597
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2022A Generalist Agentcited by 808
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  12. 2018Behavioral Cloning from Observationcited by 492
18 more
  1. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  2. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  4. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  5. 2022Multi-Game Decision Transformerscited by 214
  6. 2022Online Decision Transformercited by 196
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  8. 2018Generative Adversarial Imitation from Observationcited by 184
  9. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  10. 2020An Overview of Deep Semi-Supervised Learningcited by 121
  11. 2017Third-Person Imitation Learningcited by 103
  12. 2019Recent Advances in Imitation Learning from Observationcited by 93
  13. 2017Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learningcited by 85
  14. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 82
  15. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  16. 2022Exploration via Elliptical Episodic Bonusescited by 51
  17. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  18. 2019Learning Predictive Models From Observation and Interactioncited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 11; 2025: 7; 2026: 3.

28 papers.

Sort:

By area:

  1. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  2. 2023Learning to Act without Actionscited by 121
  3. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  4. 2023Guided Flows for Generative Modeling and Decision Makingcited by 101
  5. 2023Diffusion Models for Black-Box Optimizationcited by 76
  6. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 44
  7. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 42
  8. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  9. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  10. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  11. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 11
  12. 2024When to Sense and Control? A Time-adaptive Approach for Continuous-Time RLcited by 9
16 more
  1. 2023Decision Stacks: Flexible Reinforcement Learning via Modular Generative Modelscited by 8
  2. 2024PRIME: Scaffolding Manipulation Tasks with Behavior Primitives for Data-Efficient Imitation Learningcited by 8
  3. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  4. 2024Robust Semi-Supervised Learning in Open Environmentscited by 8
  5. 2024GARLIC: GPT-Augmented Reinforcement Learning with Intelligent Control for Vehicle Dispatchingcited by 5
  6. 2024Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution Viewcited by 3
  7. 2026Vision-Language Models Unlock Task-Centric Latent Actionscited by 3
  8. 2024Imitation Learning with Limited Actions via Diffusion Planners and Deep Koopman Controllerscited by 2
  9. 2025Offline Action-Free Learning of Ex-BMDPs by Comparing Diverse Datasetscited by 2
  10. 2025DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitionscited by 2
  11. 2025MAGIK: Mapping to Analogous Goals via Imagination-enabled Knowledge Transfercited by 1
  12. 2025SketchPlan: Diffusion Based Drone Planning From Human Sketchescited by 1
  13. 2025Generalized Linear Markov Decision Processcited by 0
  14. 2025Context Representation via Action-Free Transformer encoder-decoder for Meta Reinforcement Learningcited by 0
  15. 2026Identifying Latent Actions and Dynamics from Offline Data via Demonstrator Diversitycited by 0
  16. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  2. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  3. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  4. 2025Semi-pessimistic Reinforcement Learning
  5. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning
  6. 2024Off-dynamics Conditional Diffusion Planners
  7. 2026When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?
  8. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning