# Learning Dynamic Belief Graphs to Generalize on Text-Based Games

arXiv: https://arxiv.org/abs/2002.09127 · HTML: https://mlsanity.com/paper/2002.09127/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2020.

## Built on by (most cited first, 30 of 49)

- [Can Language Models Solve Graph Problems in Natural Language?](https://mlsanity.com/paper/2305.10037/) (2023)
- [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark](https://mlsanity.com/paper/2304.03279/) (2023)
- [AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents](https://mlsanity.com/paper/2407.04363/) (2024)
- [Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker](https://mlsanity.com/paper/2306.00924/) (2023)
- [What Would Jiminy Cricket Do? Towards Agents That Behave Morally](https://mlsanity.com/paper/2110.13136/) (2021)
- [Can Language Models Serve as Text-Based World Simulators?](https://mlsanity.com/paper/2406.06485/) (2024)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Aligning to Social Norms and Values in Interactive Narratives](https://mlsanity.com/paper/2205.01975/) (2022)
- [Learning Knowledge Graph-based World Models of Textual Environments](https://mlsanity.com/paper/2106.09608/) (2021)
- [Neuro-Symbolic Reinforcement Learning with First-Order Logic](https://mlsanity.com/paper/2110.10963/) (2021)
- [Introspective Tips: Large Language Model for In-Context Decision Making](https://mlsanity.com/paper/2305.11598/) (2023)
- [Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities](https://mlsanity.com/paper/2506.18019/) (2025)
- [Monte Carlo Planning with Large Language Model for Text-Based Game Agents](https://mlsanity.com/paper/2504.16855/) (2025)
- [A Systematic Survey of Text Worlds as Embodied Natural Language Environments](https://mlsanity.com/paper/2107.04132/) (2021)
- [Process-Level Representation of Scientific Protocols with Interactive Annotation](https://mlsanity.com/paper/2101.10244/) (2021)
- [Inherently Explainable Reinforcement Learning in Natural Language](https://mlsanity.com/paper/2112.08907/) (2021)
- [Towards Socially Intelligent Agents with Mental State Transition and Human Utility](https://mlsanity.com/paper/2103.07011/) (2021)
- [A Survey of Text Games for Reinforcement Learning informed by Natural Language](https://mlsanity.com/paper/2109.09478/) (2021)
- [Learning to Follow Instructions in Text-Based Games](https://mlsanity.com/paper/2211.04591/) (2022)
- [Modeling Worlds in Text](https://mlsanity.com/paper/2106.09578/) (2021)
- [How to Query Language Models?](https://mlsanity.com/paper/2108.01928/) (2021)
- [Generalization in Text-based Games via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2109.09968/) (2021)
- [Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning](https://mlsanity.com/paper/2211.00247/) (2022)
- [Perceiving the World: Question-guided Reinforcement Learning for Text-based Games](https://mlsanity.com/paper/2204.09597/) (2022)
- [Asking for Knowledge: Training RL Agents to Query External Knowledge Using Language](https://mlsanity.com/paper/2205.06111/) (2022)
- [Behavior Cloned Transformers are Neurosymbolic Reasoners](https://mlsanity.com/paper/2210.07382/) (2022)
- [Situated Dialogue Learning through Procedural Environment Generation](https://mlsanity.com/paper/2110.03262/) (2021)
- [Can LLMs perform structured graph reasoning?](https://mlsanity.com/paper/2402.01805/) (2024)
- [GraphThought: Graph Combinatorial Optimization with Thought Generation](https://mlsanity.com/paper/2502.11607/) (2025)
- [LOA: Logical Optimal Actions for Text-based Interaction Games](https://mlsanity.com/paper/2110.10973/) (2021)

## Abstract (arXiv)

Playing text-based games requires skills in processing natural language and sequential decision making. Achieving human-level performance on text-based games remains an open challenge, and prior research has largely relied on hand-crafted structured representations and heuristics. In this work, we investigate how an agent can plan and generalize in text-based games using graph-structured representations learned end-to-end from raw text. We propose a novel graph-aided transformer agent (GATA) that infers and updates latent belief graphs during planning to enable effective action selection by capturing the underlying game dynamics. GATA is trained using a combination of reinforcement and self-supervised learning. Our work demonstrates that the learned graph-based representations help agents converge to better policies than their text-only counterparts and facilitate effective generalization across game configurations. Experiments on 500+ unique games from the TextWorld suite show that our best agent outperforms text-based baselines by an average of 24.2%.