mlsanityData to 14 Oct 2026

Accelerating exploration and representation learning with offline pre-training

arXiv 2304.00046 ↗First posted 31 Mar 2023cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Sequential decision-making agents struggle with long horizon tasks, since solving them requires multi-step reasoning. Most reinforcement learning (RL) algorithms address this challenge by improved credit assignment, introducing memory capability, altering the agent's intrinsic motivation (i.e. exploration) or its worldview (i.e. knowledge representation). Many of these components could be learned from offline data. In this work, we follow the hypothesis that exploration and representation learning can be improved by separately learning two different models from a single offline dataset. We show that learning a state representation using noise-contrastive estimation and a model of auxiliary reward separately from a single collection of human demonstrations can significantly improve the sample efficiency on the challenging NetHack benchmark. We also ablate various components of our experimental setting and highlight crucial insights.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 44 of its 53 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

44 of 53 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2016Generative Adversarial Imitation Learningcited by 1,597
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2018Exploration by Random Network Distillationcited by 993
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  9. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  10. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  11. 2019On Variational Bounds of Mutual Informationcited by 559
  12. 2018Behavioral Cloning from Observationcited by 492
32 more
  1. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  2. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  3. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  4. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  5. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  6. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  7. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  8. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  9. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  10. 2020The NetHack Learning Environmentcited by 198
  11. 2018Generative Adversarial Imitation from Observationcited by 184
  12. 2021Reinforcement Learning with Prototypical Representationscited by 179
  13. 2019Unsupervised State Representation Learning in Ataricited by 144
  14. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  15. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  16. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 105
  17. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  18. 2022Deep Hierarchical Planning from Pixelscited by 101
  19. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 91
  20. 2022A data-driven approach for learning to control computerscited by 88
  21. 2020Deep Reinforcement and InfoMax Learningcited by 79
  22. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  23. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  24. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  25. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 45
  26. 2021Heuristic-Guided Reinforcement Learningcited by 42
  27. 2021Podracer architectures for scalable Reinforcement Learningcited by 38
  28. 2020MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Explorationcited by 25
  29. 2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
  30. 2022Insights From the NeurIPS 2021 NetHack Challengecited by 20
  31. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  32. 2021Imitation by Predicting Observationscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 1; 2026: 1.

7 papers.

  1. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 39
  2. 2023Scaling Laws for Imitation Learning in Single-Agent Gamescited by 11
  3. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measurescited by 3
  4. 2024Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidancecited by 2
  5. 2025Disentangling Exploration of Large Language Models by Optimal Exploitationcited by 1
  6. 2023Value Explicit Pretraining for Learning Transferable Representationscited by 0
  7. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measurescited by 0