# Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals

arXiv: https://arxiv.org/abs/2302.04449 · HTML: https://mlsanity.com/paper/2302.04449/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 72 references that are in the map.
- First posted on arXiv 9 Feb 2023.

## Builds on (most cited first, 30 of 54)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)

## Built on by (most cited first, 30 of 45)

- [Text2Reward: Reward Shaping with Language Models for Reinforcement Learning](https://mlsanity.com/paper/2309.11489/) (2023)
- [SmartPlay: A Benchmark for LLMs as Intelligent Agents](https://mlsanity.com/paper/2310.01557/) (2023)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [LLaRA: Supercharging Robot Learning Data for Vision-Language Policy](https://mlsanity.com/paper/2406.20095/) (2024)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models](https://mlsanity.com/paper/2310.15127/) (2023)
- [OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environments](https://mlsanity.com/paper/2306.08649/) (2023)
- [Interpretable Concept Bottlenecks to Align Reinforcement Learning Agents](https://mlsanity.com/paper/2401.05821/) (2024)
- [Semantic HELM: A Human-Readable Memory for Reinforcement Learning](https://mlsanity.com/paper/2306.09312/) (2023)
- [Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication](https://mlsanity.com/paper/2409.17348/) (2024)
- [AgentKit: Structured LLM Reasoning with Dynamic Graphs](https://mlsanity.com/paper/2404.11483/) (2024)
- [Language-Guided World Models: A Model-Based Approach to AI Control](https://mlsanity.com/paper/2402.01695/) (2024)
- [SPRING: Studying the Paper and Reasoning to Play Games](https://mlsanity.com/paper/2305.15486/) (2023)
- [How Well Can Vision Language Models See Image Details?](https://mlsanity.com/paper/2408.03940/) (2024)
- [Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback](https://mlsanity.com/paper/2410.23022/) (2024)
- [Towards Generalist Robots: A Promising Paradigm via Generative Simulation](https://mlsanity.com/paper/2305.10455/) (2023)
- [Enhancing Agent Learning through World Dynamics Modeling](https://mlsanity.com/paper/2407.17695/) (2024)
- [Intrinsic Language-Guided Exploration for Complex Long-Horizon Robotic Manipulation Tasks](https://mlsanity.com/paper/2309.16347/) (2023)
- [Predicting and Understanding Human Action Decisions: Insights from Large Language Models and Cognitive Instance-Based Learning](https://mlsanity.com/paper/2407.09281/) (2024)
- [Atari-GPT: Benchmarking Multimodal Large Language Models as Low-Level Policies in Atari Games](https://mlsanity.com/paper/2408.15950/) (2024)
- [Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RL](https://mlsanity.com/paper/2409.12798/) (2024)
- [Zero-shot Model-based Reinforcement Learning using Large Language Models](https://mlsanity.com/paper/2410.11711/) (2024)
- [Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence](https://mlsanity.com/paper/2402.02701/) (2024)
- [SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding](https://mlsanity.com/paper/2603.09036/) (2026)
- [Towards Socially and Morally Aware RL agent: Reward Design With LLM](https://mlsanity.com/paper/2401.12459/) (2024)
- [Read to Play (R2-Play): Decision Transformer with Multimodal Game Instruction](https://mlsanity.com/paper/2402.04154/) (2024)
- [Interpretable end-to-end Neurosymbolic Reinforcement Learning agents](https://mlsanity.com/paper/2410.14371/) (2024)
- [LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models](https://mlsanity.com/paper/2505.15293/) (2025)
- [From Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided Knowledge](https://mlsanity.com/paper/2410.01458/) (2024)
- [Data to Defense: The Role of Curation in Customizing LLMs Against Jailbreaking Attacks](https://mlsanity.com/paper/2410.02220/) (2024)

## Abstract (arXiv)

High sample complexity has long been a challenge for RL. On the other hand, humans learn to perform tasks not only from interaction or demonstrations, but also by reading unstructured text documents, e.g., instruction manuals. Instruction manuals and wiki pages are among the most abundant data that could inform agents of valuable features and policies or task-specific environmental dynamics and reward structures. Therefore, we hypothesize that the ability to utilize human-written instruction manuals to assist learning policies for specific tasks should lead to a more efficient and better-performing agent. We propose the Read and Reward framework. Read and Reward speeds up RL algorithms on Atari games by reading manuals released by the Atari game developers. Our framework consists of a QA Extraction module that extracts and summarizes relevant information from the manual and a Reasoning module that evaluates object-agent interactions based on information from the manual. An auxiliary reward is then provided to a standard A2C RL agent, when interaction is detected. Experimentally, various RL algorithms obtain significant improvement in performance and training speed when assisted by our design.