# EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning

arXiv: https://arxiv.org/abs/2403.10692 · HTML: https://mlsanity.com/paper/2403.10692/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 23 of 23)

- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [Learning Explanatory Rules from Noisy Data](https://mlsanity.com/paper/1711.04574/) (2017)
- [End-to-End Differentiable Proving](https://mlsanity.com/paper/1705.11040/) (2017)
- [SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning](https://mlsanity.com/paper/1811.00090/) (2018)
- [Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning](https://mlsanity.com/paper/1809.02121/) (2018)
- [Language Understanding for Text-based Games Using Deep Reinforcement Learning](https://mlsanity.com/paper/1506.08941/) (2015)
- [Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning](https://mlsanity.com/paper/1812.01628/) (2018)
- [PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making](https://mlsanity.com/paper/1804.07779/) (2018)
- [Learning Dynamic Belief Graphs to Generalize on Text-Based Games](https://mlsanity.com/paper/2002.09127/) (2020)
- [Constraint Answer Set Programming without Grounding](https://mlsanity.com/paper/1804.11162/) (2018)
- [Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines](https://mlsanity.com/paper/2010.03790/) (2020)
- [Neuro-Symbolic Reinforcement Learning with First-Order Logic](https://mlsanity.com/paper/2110.10963/) (2021)
- [NAIL: A General Interactive Fiction Agent](https://mlsanity.com/paper/1902.04259/) (2019)
- [LeDeepChef: Deep Reinforcement Learning Agent for Families of Text-Based Games](https://mlsanity.com/paper/1909.01646/) (2019)
- [A New Algorithm to Automate Inductive Learning of Default Theories](https://mlsanity.com/paper/1707.02693/) (2017)
- [Justifications for Goal-Directed Constraint Answer Set Programming](https://mlsanity.com/paper/2009.10238/) (2020)
- [AUTO-DISCERN: Autonomous Driving Using Common Sense Reasoning](https://mlsanity.com/paper/2110.13606/) (2021)
- [Knowledge-driven Natural Language Understanding of English Text and its Applications](https://mlsanity.com/paper/2101.11707/) (2021)
- [Automated Interactive Domain-Specific Conversational Agents that Understand Human Dialogs](https://mlsanity.com/paper/2303.08941/) (2023)
- [Behavior Cloned Transformers are Neurosymbolic Reasoners](https://mlsanity.com/paper/2210.07382/) (2022)
- [Case-based Reasoning for Better Generalization in Textual Reinforcement Learning](https://mlsanity.com/paper/2110.08470/) (2021)
- [Learning Symbolic Rules over Abstract Meaning Representations for Textual Reinforcement Learning](https://mlsanity.com/paper/2307.02689/) (2023)
- [An ASP-based Approach to Answering Natural Language Questions for Texts](https://mlsanity.com/paper/2112.11241/) (2021)

## Built on by (most cited first, 3 of 3)

- [AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents](https://mlsanity.com/paper/2407.04363/) (2024)
- [Cross-Environment Neural Reranking for Sample-Efficient Action Selection in Text-Based Agents](https://mlsanity.com/paper/2606.02204/) (2026)
- [LLM-Augmented Symbolic Reinforcement Learning with Landmark-Based Task Decomposition](https://mlsanity.com/paper/2410.01929/) (2024)

## Abstract (arXiv)

Text-based games (TBGs) have emerged as an important collection of NLP tasks, requiring reinforcement learning (RL) agents to combine natural language understanding with reasoning. A key challenge for agents attempting to solve such tasks is to generalize across multiple games and demonstrate good performance on both seen and unseen objects. Purely deep-RL-based approaches may perform well on seen objects; however, they fail to showcase the same performance on unseen objects. Commonsense-infused deep-RL agents may work better on unseen data; unfortunately, their policies are often not interpretable or easily transferable. To tackle these issues, in this paper, we present EXPLORER which is an exploration-guided reasoning agent for textual reinforcement learning. EXPLORER is neurosymbolic in nature, as it relies on a neural module for exploration and a symbolic module for exploitation. It can also learn generalized symbolic policies and perform well over unseen data. Our experiments show that EXPLORER outperforms the baseline agents on Text-World cooking (TW-Cooking) and Text-World Commonsense (TWC) games.