mlsanityData to 14 Oct 2026

APS: Active Pretraining with Successor Features

arXiv 2108.13956 ↗First posted 31 Aug 2021cs.LGcited by 112

Not on the map. Why?

Cited by 29 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a new unsupervised pretraining objective for reinforcement learning. During the unsupervised reward-free pretraining phase, the agent maximizes mutual information between tasks and states induced by the policy. Our key contribution is a novel lower bound of this intractable quantity. We show that by reinterpreting and combining variational successor features~citepHansen2020Fast with nonparametric entropy maximization~citepliu2021behavior, the intractable mutual information can be efficiently optimized. The proposed method Active Pretraining with Successor Feature (APS) explores the environment via nonparametric entropy maximization, and the explored data can be efficiently leveraged to learn behavior by variational successor features. APS addresses the limitations of existing mutual information maximization based and entropy maximization based unsupervised RL, and combines the best of both worlds. When evaluated on the Atari 100k data-efficiency benchmark, our approach significantly outperforms previous methods combining unsupervised pretraining with task-specific finetuning.

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). Builds on 41 of its 64 references that are in the map. First posted on arXiv 31 Aug 2021.

Builds on

41 of 64 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  10. 2017Hindsight Experience Replaycited by 1,136
  11. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
29 more
  1. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  2. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  3. 2019Solving Rubik's Cube with a Robot Handcited by 734
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  5. 2019Model-Based Reinforcement Learning for Ataricited by 502
  6. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  7. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  8. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  9. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  10. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  11. 2016Variational Intrinsic Controlcited by 267
  12. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  13. 2020Never Give Up: Learning Directed Exploration Strategiescited by 212
  14. 2019Efficient Exploration via State Marginal Matchingcited by 188
  15. 2021Reinforcement Learning with Prototypical Representationscited by 179
  16. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  17. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
  18. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  19. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  20. 2019When to use parametric models in reinforcement learning?cited by 118
  21. 2018Count-Based Exploration with the Successor Representationcited by 112
  22. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  23. 2018Universal Successor Features Approximatorscited by 103
  24. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  25. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  26. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  27. 2016Deep Successor Reinforcement Learningcited by 72
  28. 2020Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimatecited by 21
  29. 2016VIME: Variational Information Maximizing Explorationcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 34; 2023: 23; 2024: 21; 2025: 15; 2026: 13.

112 papers.

Sort:

By area:

  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  2. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  3. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  4. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  5. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  6. 2023XSkill: Cross Embodiment Skill Discoverycited by 75
  7. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 68
  8. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 62
  9. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
  10. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 54
  11. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  12. 2022Masked Autoencoding for Scalable and Generalizable Decision Makingcited by 45
38 more
  1. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  2. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  3. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  4. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  5. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 30
  6. 2022Continuous MDP Homomorphisms and Homomorphic Policy Gradientcited by 27
  7. 2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
  8. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 25
  9. 2022Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfercited by 22
  10. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  11. 2021Dynamic Bottleneck for Robust Self-Supervised Explorationcited by 21
  12. 2022Mastering the Unsupervised Reinforcement Learning Benchmark from Pixelscited by 21
  13. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  14. 2024Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learningcited by 21
  15. 2025Fast Adaptation with Behavioral Foundation Modelscited by 21
  16. 2023Explore to Generalize in Zero-Shot RLcited by 20
  17. 2021Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reachingcited by 19
  18. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  19. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  20. 2023Pre-Training and Fine-Tuning Generative Flow Networkscited by 17
  21. 2024Language Guided Skill Discoverycited by 16
  22. 2023Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generationcited by 15
  23. 2021A First-Occupancy Representation for Reinforcement Learningcited by 14
  24. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teacherscited by 14
  25. 2024Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Expertscited by 14
  26. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  27. 2022k-Means Maximum Entropy Explorationcited by 11
  28. 2022A Mixture of Surprises for Unsupervised Reinforcement Learningcited by 9
  29. 2023Policy Gradient Methods in the Presence of Symmetries and State Abstractionscited by 9
  30. 2023Variational Curriculum Reinforcement Learning for Unsupervised Discovery of Skillscited by 9
  31. 2024PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learningcited by 9
  32. 2022EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Modelcited by 8
  33. 2023Learning to Discover Skills through Guidancecited by 8
  34. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 8
  35. 2024Do's and Don'ts: Learning Desirable Skills with Instruction Videoscited by 8
  36. 2025Epistemically-guided forward-backward explorationcited by 8
  37. 2022Palm up: Playing in the Latent Manifold for Unsupervised Pretrainingcited by 7
  38. 2022A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Processcited by 7

50 most cited of 112 All 112 (JSON)