mlsanityData to 7 Oct 2026

Asking for Knowledge: Training RL Agents to Query External Knowledge Using Language

arXiv 2205.06111 ↗First posted 12 May 2022cs.AI cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

To solve difficult tasks, humans ask questions to acquire knowledge from external sources. In contrast, classical reinforcement learning agents lack such an ability and often resort to exploratory behavior. This is exacerbated as few present-day environments support querying for knowledge. In order to study how agents can be taught to query external knowledge via language, we first introduce two new environments: the grid-world-based Q-BabyAI and the text-based Q-TextWorld. In addition to physical interactions, an agent can query an external knowledge source specialized for these environments to gather information. Second, we propose the "Asking for Knowledge" (AFK) agent, which learns to generate language commands to query for meaningful knowledge that helps solve the tasks. AFK leverages a non-parametric memory, a pointer mechanism and an episodic exploration bonus to tackle (1) irrelevant information, (2) a large query language space, (3) delayed reward for making meaningful queries. Extensive experiments demonstrate that the AFK agent outperforms recent baselines on the challenging Q-BabyAI and Q-TextWorld environments.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 48 of its 73 references that are in the map. First posted on arXiv 12 May 2022.

Builds on

48 of 73 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2016Layer Normalizationcited by 5,579
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  8. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2017Deep Setscited by 1,394
  11. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
36 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  3. 2017Deep Reinforcement Learning that Matterscited by 727
  4. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  5. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  6. 2014Memory Networkscited by 354
  7. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  8. 2017Count-Based Exploration with Neural Density Modelscited by 255
  9. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  10. 2017Learning to Ask: Neural Question Generation for Reading Comprehensioncited by 186
  11. 2017Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learningcited by 161
  12. 2020Answering Complex Open-Domain Questions with Multi-Hop Dense Retrievalcited by 160
  13. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  14. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  15. 2018Episodic Curiosity through Reachabilitycited by 109
  16. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  17. 2020Graph Constrained Reinforcement Learning for Natural Language Action Spacescited by 80
  18. 2017Learning how to Active Learn: A Deep Reinforcement Learning Approachcited by 79
  19. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  20. 2020Learning Dynamic Belief Graphs to Generalize on Text-Based Gamescited by 49
  21. 2017Ask the Right Questions: Active Question Reformulation with Reinforcement Learningcited by 44
  22. 2019Metalearned Neural Memorycited by 44
  23. 2018GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithmscited by 40
  24. 2017Task-Oriented Query Reformulation with Reinforcement Learningcited by 39
  25. 2019Multi-Hop Paragraph Retrieval for Open-Domain Question Answeringcited by 34
  26. 2019RTFM: Generalising to Novel Environment Dynamics via Readingcited by 33
  27. 2017Machine Comprehension by Text-to-Text Neural Question Generationcited by 32
  28. 2017Learning Algorithms for Active Learningcited by 30
  29. 2019Ask to Learn: A Study on Curiosity-driven Question Generationcited by 15
  30. 2018Two can play this Game: Visual Dialog with Discriminative Question Generation and Answeringcited by 13
  31. 2021Reinforcement Learning with External Knowledge by using Logical Neural Networkscited by 10
  32. 2021SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agentscited by 7
  33. 2017Reinforcement Learning with External Knowledge and Two-Stage Q-functions for Predicting Popular Reddit Threadscited by 3
  34. 2019Playing log(N)-Questions over Sentencescited by 2
  35. 2021Interactive Machine Comprehension with Dynamic Knowledge Graphscited by 2
  36. 2017Deep Reinforcement Learning Boosted by External Knowledgecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 2; 2025: 1; 2026: 2.

10 papers.

  1. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  2. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  3. 2024Language-guided Skill Learning with Temporal Variational Inferencecited by 11
  4. 2022Behavior Cloned Transformers are Neurosymbolic Reasonerscited by 10
  5. 2023Architext: Language-Driven Generative Architecture Designcited by 5
  6. 2026MINT: Minimal Information Neuro-Symbolic Tree for Objective-Driven Knowledge-Gap Reasoning and Active Elicitationcited by 4
  7. 2023Improve the efficiency of deep reinforcement learning through semantic exploration guided by natural languagecited by 1
  8. 2024Getting By Goal Misgeneralization With a Little Help From a Mentorcited by 1
  9. 2025How2: How to learn from procedural How-to questionscited by 1
  10. 2026Knowledge Boundary Discovery for Large Language Modelscited by 1