mlsanityData to 13 Oct 2026

Query The Agent: Improving sample efficiency through epistemic uncertainty estimation

arXiv 2210.02585 ↗First posted 5 Oct 2022cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Curricula for goal-conditioned reinforcement learning agents typically rely on poor estimates of the agent's epistemic uncertainty or fail to consider the agents' epistemic uncertainty altogether, resulting in poor sample efficiency. We propose a novel algorithm, Query The Agent (QTA), which significantly improves sample efficiency by estimating the agent's epistemic uncertainty throughout the state space and setting goals in highly uncertain areas. Encouraging the agent to collect data in highly uncertain states allows the agent to improve its estimation of the value function rapidly. QTA utilizes a novel technique for estimating epistemic uncertainty, Predictive Uncertainty Networks (PUN), to allow QTA to assess the agent's uncertainty in all previously observed states. We demonstrate that QTA offers decisive sample efficiency improvements over preexisting methods.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 25 of its 29 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

25 of 29 references.

References · 25

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,902
  2. 2015Continuous control with deep reinforcement learningcited by 3,290
  3. 2017Hindsight Experience Replaycited by 1,104
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,093
  5. 2015Prioritized Experience Replaycited by 1,078
  6. 2018Exploration by Random Network Distillationcited by 971
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  8. 2016Deep Exploration via Bootstrapped DQNcited by 520
  9. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 442
  10. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  11. 2019Challenges of Real-World Reinforcement Learningcited by 285
  12. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 279
13 more
  1. 2019Self-Supervised Exploration via Disagreementcited by 264
  2. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  3. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 254
  4. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  5. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 115
  6. 2018Count-Based Exploration with the Successor Representationcited by 112
  7. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  8. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  9. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  10. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  11. 2021Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Reviewcited by 34
  12. 2021DriverGym: Democratising Reinforcement Learning for Autonomous Drivingcited by 23
  13. 2020Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planningcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Uncertainty Prioritized Experience Replaycited by 5