# Learning to Query Internet Text for Informing Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/2205.13079 · HTML: https://mlsanity.com/paper/2205.13079/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 15 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 15 of 15)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Pre-Trained Language Models for Interactive Decision-Making](https://mlsanity.com/paper/2202.01771/) (2022)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [A Survey of Reinforcement Learning Informed by Natural Language](https://mlsanity.com/paper/1906.03926/) (2019)
- [MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research](https://mlsanity.com/paper/2109.13202/) (2021)
- [Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text](https://mlsanity.com/paper/2005.09382/) (2020)
- [Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.07393/) (2021)
- [How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds](https://mlsanity.com/paper/2006.07409/) (2020)
- [RTFM: Generalising to Novel Environment Dynamics via Reading](https://mlsanity.com/paper/1910.08210/) (2019)
- [Playing Text-Based Games with Common Sense](https://mlsanity.com/paper/2012.02757/) (2020)
- [Learning to Generalize for Sequential Decision Making](https://mlsanity.com/paper/2010.02229/) (2020)

## Built on by (most cited first, 3 of 3)

- [Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling](https://mlsanity.com/paper/2301.12050/) (2023)
- [Language-Guided World Models: A Model-Based Approach to AI Control](https://mlsanity.com/paper/2402.01695/) (2024)
- [Probing for Consciousness in Machines](https://mlsanity.com/paper/2411.16262/) (2024)

## Abstract (arXiv)

Generalization to out of distribution tasks in reinforcement learning is a challenging problem. One successful approach improves generalization by conditioning policies on task or environment descriptions that provide information about the current transition or reward functions. Previously, these descriptions were often expressed as generated or crowd sourced text. In this work, we begin to tackle the problem of extracting useful information from natural language found in the wild (e.g. internet forums, documentation, and wikis). These natural, pre-existing sources are especially challenging, noisy, and large and present novel challenges compared to previous approaches. We propose to address these challenges by training reinforcement learning agents to learn to query these sources as a human would, and we experiment with how and when an agent should query. To address the textithow, we demonstrate that pretrained QA models perform well at executing zero-shot queries in our target domain. Using information retrieved by a QA model, we train an agent to learn textitwhen it should execute queries. We show that our method correctly learns to execute queries to maximize reward in a reinforcement learning setting.