mlsanityData to 14 Oct 2026

Guiding Online Reinforcement Learning with Action-Free Offline Pretraining

arXiv 2301.12876 ↗First posted 30 Jan 2023cs.LG cs.AIcited by 11

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline RL methods have been shown to reduce the need for environment interaction by training agents using offline collected episodes. However, these methods typically require action information to be logged during data collection, which can be difficult or even impossible in some practical cases. In this paper, we investigate the potential of using action-free offline datasets to improve online reinforcement learning, name this problem Reinforcement Learning with Action-Free Offline Pretraining (AFP-RL). We introduce Action-Free Guide (AF-Guide), a method that guides online training by extracting knowledge from action-free offline datasets. AF-Guide consists of an Action-Free Decision Transformer (AFDT) implementing a variant of Upside-Down Reinforcement Learning. It learns to plan the next states from the offline dataset, and a Guided Soft Actor-Critic (Guided SAC) that learns online with guidance from AFDT. Experimental results show that AF-Guide can improve sample efficiency and performance in online training thanks to the knowledge from the action-free offline dataset. Code is available at https://github.com/Vision-CAIR/AF-Guide.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 40 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

28 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 690
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2018Behavioral Cloning from Observationcited by 492
  10. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 331
  11. 2020Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Predictioncited by 325
  12. 2021Multimodal Motion Prediction with Stacked Transformerscited by 208
12 more
  1. 2018Generative Adversarial Imitation from Observationcited by 184
  2. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  3. 2020A Spatio-temporal Transformer for 3D Human Motion Predictioncited by 133
  4. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  5. 2018Zero-Shot Visual Imitationcited by 117
  6. 2015On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Modelscited by 55
  7. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreementcited by 46
  8. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  9. 2022Social-Implicit: Rethinking Trajectory Prediction Evaluation and The Effectiveness of Implicit Maximum Likelihood Estimationcited by 40
  10. 2020Novelty Search in Representational Space for Sample Efficient Explorationcited by 26
  11. 2018One Big Net For Everythingcited by 16
  12. 2022Upside-Down Reinforcement Learning Can Diverge in Stochastic Environments With Episodic Resetscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 0; 2026: 2.

11 papers.

  1. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  2. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  3. 2023Learning from Visual Observation via Offline Pretrained State-to-Go Transformercited by 14
  4. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
  5. 2023Learning to Identify Critical States for Reinforcement Learning from Videoscited by 9
  6. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  7. 2024NOLO: Navigate Only Look Oncecited by 3
  8. 2024Snapshot Reinforcement Learning: Leveraging Prior Trajectories for Efficiencycited by 0
  9. 2024A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learningcited by 0
  10. 2026Action-Free Offline-to-Online RL via Discretised State Policiescited by 0
  11. 2026Reinforcement Learning: From Algorithms To Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Minimalist Approach to Offline Reinforcement Learning
  2. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  3. 2023Boosting Offline Reinforcement Learning with Action Preference Query
  4. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  5. 2025A Clean Slate for Offline Reinforcement Learning
  6. 2022Leveraging Offline Data in Online Reinforcement Learning
  7. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance
  8. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model