mlsanityData to 7 Oct 2026

Reinforcement Learning with Unsupervised Auxiliary Tasks

arXiv 1611.05397 ↗First posted 16 Nov 2016cs.LG cs.NEcited by 400

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning agents have achieved state-of-the-art results by directly maximising cumulative reward. However, environments contain a much wider variety of possible training signals. In this paper, we introduce an agent that also maximises many other pseudo-reward functions simultaneously by reinforcement learning. All of these tasks share a common representation that, like unsupervised learning, continues to develop in the absence of extrinsic rewards. We also introduce a novel mechanism for focusing this representation upon extrinsic rewards, so that learning can rapidly adapt to the most relevant aspects of the actual task. Our agent significantly outperforms the previous state-of-the-art on Atari, averaging 880% expert human performance, and a challenging suite of first-person, three-dimensional Labyrinth tasks leading to a mean speedup in learning of 10× and averaging 87% expert human performance on Labyrinth.

Key facts. Cited by 400 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 5; 2019: 2; 2020: 2; 2021: 114; 2022: 81; 2023: 67; 2024: 46; 2025: 45; 2026: 38.

400 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  4. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  5. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  6. 2022Masked Visual Pre-training for Motor Controlcited by 246
  7. 2021Reinforcement Learning with Prototypical Representationscited by 179
  8. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  9. 2022Masked World Models for Visual Controlcited by 169
  10. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  11. 2021The Benchmark Lotterycited by 133
  12. 2021Proof Artifact Co-training for Theorem Proving with Language Modelscited by 130
38 more
  1. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  2. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  3. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  4. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  5. 2021SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networkscited by 108
  6. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  7. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  8. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  9. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 97
  10. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  11. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  12. 2022Offline Visual Representation Learning for Embodied Navigationcited by 74
  13. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
  14. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  15. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  16. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  17. 2021Learning Auxiliary Monocular Contexts Helps Monocular 3D Object Detectioncited by 66
  18. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  19. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 65
  20. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 61
  21. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 60
  22. 2025Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQLcited by 59
  23. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  24. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  25. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  26. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  27. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 52
  28. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  29. 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
  30. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 46
  31. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  32. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  33. 2021Curious Representation Learning for Embodied Intelligencecited by 43
  34. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  35. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
  36. 2021CompilerGym: Robust, Performant Compiler Optimization Environments for AI Researchcited by 41
  37. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  38. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41

50 most cited of 400 All 400 (JSON)