# Deep Reinforcement Learning with a Natural Language Action Space

arXiv: https://arxiv.org/abs/1511.04636 · HTML: https://mlsanity.com/paper/1511.04636/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2015.

## Built on by (most cited first, 30 of 88)

- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark](https://mlsanity.com/paper/2304.03279/) (2023)
- [SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks](https://mlsanity.com/paper/2305.17390/) (2023)
- [CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization](https://mlsanity.com/paper/2310.10134/) (2023)
- [What Would Jiminy Cricket Do? Towards Agents That Behave Morally](https://mlsanity.com/paper/2110.13136/) (2021)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [Value Augmented Sampling for Language Model Alignment and Personalization](https://mlsanity.com/paper/2405.06639/) (2024)
- [Agent-Oriented Planning in Multi-Agent Systems](https://mlsanity.com/paper/2410.02189/) (2024)
- [Recomposing the Reinforcement Learning Building Blocks with Hypernetworks](https://mlsanity.com/paper/2106.06842/) (2021)
- [Aligning to Social Norms and Values in Interactive Narratives](https://mlsanity.com/paper/2205.01975/) (2022)
- [Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents](https://mlsanity.com/paper/2305.02412/) (2023)
- [Survey on reinforcement learning for language processing](https://mlsanity.com/paper/2104.05565/) (2021)
- [Learning Knowledge Graph-based World Models of Textual Environments](https://mlsanity.com/paper/2106.09608/) (2021)
- [Augmenting Autotelic Agents with Large Language Models](https://mlsanity.com/paper/2305.12487/) (2023)
- [Language Models are Few-Shot Butlers](https://mlsanity.com/paper/2104.07972/) (2021)
- [ChatShop: Interactive Information Seeking with Language Agents](https://mlsanity.com/paper/2404.09911/) (2024)
- [Multi-Stage Episodic Control for Strategic Exploration in Text Games](https://mlsanity.com/paper/2201.01251/) (2022)
- [Monte Carlo Planning with Large Language Model for Text-Based Game Agents](https://mlsanity.com/paper/2504.16855/) (2025)
- [Pre-trained Language Models as Prior Knowledge for Playing Text-based Games](https://mlsanity.com/paper/2107.08408/) (2021)
- [Reading and Acting while Blindfolded: The Need for Semantics in Text Game Agents](https://mlsanity.com/paper/2103.13552/) (2021)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Contrastive Reinforcement Learning of Symbolic Reasoning Domains](https://mlsanity.com/paper/2106.09146/) (2021)
- [A Systematic Survey of Text Worlds as Embodied Natural Language Environments](https://mlsanity.com/paper/2107.04132/) (2021)
- [Training Value-Aligned Reinforcement Learning Agents Using a Normative Prior](https://mlsanity.com/paper/2104.09469/) (2021)
- [A Survey of Text Games for Reinforcement Learning informed by Natural Language](https://mlsanity.com/paper/2109.09478/) (2021)
- [Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks](https://mlsanity.com/paper/2011.02511/) (2020)
- [Learning to Follow Instructions in Text-Based Games](https://mlsanity.com/paper/2211.04591/) (2022)
- [Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations](https://mlsanity.com/paper/2405.13828/) (2024)
- [Generalization in Text-based Games via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2109.09968/) (2021)
- [Perceiving the World: Question-guided Reinforcement Learning for Text-based Games](https://mlsanity.com/paper/2204.09597/) (2022)

## Abstract (arXiv)

This paper introduces a novel architecture for reinforcement learning with deep neural networks designed to handle state and action spaces characterized by natural language, as found in text-based games. Termed a deep reinforcement relevance network (DRRN), the architecture represents action and state spaces with separate embedding vectors, which are combined with an interaction function to approximate the Q-function in reinforcement learning. We evaluate the DRRN on two popular text games, showing superior performance over other deep Q-learning architectures. Experiments with paraphrased action descriptions show that the model is extracting meaning rather than simply memorizing strings of text.