mlsanityData to 7 Oct 2026

Behavior From the Void: Unsupervised Active Pre-Training

arXiv 2103.04551 ↗First posted 8 Mar 2021cs.LGcited by 173

Not on the map. Why?

Abstract (arXiv)

We introduce a new unsupervised pre-training method for reinforcement learning called APT, which stands for Active Pre-Training. APT learns behaviors and representations by actively searching for novel states in reward-free environments. The key novel idea is to explore the environment by maximizing a non-parametric entropy computed in an abstract representation space, which avoids challenging density modeling and consequently allows our approach to scale much better in environments that have high-dimensional observations (e.g., image observations). We empirically evaluate APT by exposing task-specific reward after a long unsupervised pre-training phase. In Atari games, APT achieves human-level performance on 12 games and obtains highly competitive performance compared to canonical fully supervised RL algorithms. On DMControl suite, APT beats all baselines in terms of asymptotic performance and data efficiency and dramatically improves performance on tasks that are extremely difficult to train from scratch.

Key facts. Cited by 173 papers in the map (data through 7 Oct 2026). Builds on 53 of its 81 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

53 of 81 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2016Layer Normalizationcited by 5,579
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  10. 2013Auto-Encoding Variational Bayescited by 3,980
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
38 more
  1. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  5. 2020Mastering Atari with Discrete World Modelscited by 990
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  10. 2019Solving Rubik's Cube with a Robot Handcited by 699
  11. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  12. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  13. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  14. 2019Model-Based Reinforcement Learning for Ataricited by 495
  15. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  16. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  17. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  18. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  19. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  20. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  21. 2019Self-Supervised Exploration via Disagreementcited by 263
  22. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  23. 2016Variational Intrinsic Controlcited by 262
  24. 2017Count-Based Exploration with Neural Density Modelscited by 255
  25. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  26. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 231
  27. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  28. 2019Efficient Exploration via State Marginal Matchingcited by 186
  29. 2021Reinforcement Learning with Prototypical Representationscited by 179
  30. 2019Unsupervised State Representation Learning in Ataricited by 144
  31. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  32. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  33. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  34. 2019When to use parametric models in reinforcement learning?cited by 118
  35. 2018Count-Based Exploration with the Successor Representationcited by 112
  36. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  37. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  38. 2016Learning values across many orders of magnitudecited by 80

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 40; 2023: 42; 2024: 32; 2025: 28; 2026: 22.

173 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  3. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  4. 2022Masked World Models for Visual Controlcited by 169
  5. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  6. 2021APS: Active Pretraining with Successor Featurescited by 109
  7. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  8. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  9. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  10. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
  11. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  12. 2024Foundation Policies with Hilbert Representationscited by 89
38 more
  1. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  2. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  3. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 60
  4. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  5. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  6. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  7. 2025Self-Questioning Language Modelscited by 47
  8. 2022Masked Autoencoding for Scalable and Generalizable Decision Makingcited by 44
  9. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  10. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 42
  11. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  12. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  13. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  14. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  15. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  16. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  17. 2022Human-level Atari 200x fastercited by 28
  18. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  19. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  20. 2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
  21. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
  22. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
  23. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  24. 2025Embodied AI: From LLMs to World Modelscited by 23
  25. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  26. 2021Dynamic Bottleneck for Robust Self-Supervised Explorationcited by 21
  27. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  28. 2023Explore to Generalize in Zero-Shot RLcited by 20
  29. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  30. 2022Unsupervised Representation Learning in Deep Reinforcement Learning: A Reviewcited by 18
  31. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  32. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  33. 2023FOCUS: Object-Centric World Models for Robotics Manipulationcited by 17
  34. 2024TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representationscited by 17
  35. 2023SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learningcited by 16
  36. 2024Language Guided Skill Discoverycited by 16
  37. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
  38. 2024Global Reinforcement Learning: Beyond Linear and Convex Rewards via Submodular Semi-gradient Methodscited by 15

50 most cited of 173 All 173 (JSON)