mlsanityData to 7 Oct 2026

Learning Knowledge Graph-based World Models of Textual Environments

arXiv 2106.09608 ↗First posted 17 Jun 2021cs.LG cs.AI cs.CLcited by 31

On the map: World models → World models for language agents

Abstract (arXiv)

World models improve a learning agent's ability to efficiently operate in interactive and situated environments. This work focuses on the task of building world models of text-based game environments. Text-based games, or interactive narratives, are reinforcement learning environments in which agents perceive and interact with the world using textual natural language. These environments contain long, multi-step puzzles or quests woven through a world that is filled with hundreds of characters, locations, and objects. Our world model learns to simultaneously: (1) predict changes in the world caused by an agent's actions when representing the world as a knowledge graph; and (2) generate the set of contextually relevant natural language actions required to operate in the world. We frame this task as a Set of Sequences generation problem by exploiting the inherent structure of knowledge graphs and actions and introduce both a transformer-based multi-task architecture and a loss function to train it. A zero-shot ablation study on never-before-seen textual worlds shows that our methodology significantly outperforms existing textual world modeling techniques as well as the importance of each of our contributions.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 29 of its 49 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

29 of 49 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2017Deep Setscited by 1,394
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  11. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  12. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
17 more
  1. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  2. 2018Learning Deep Generative Models of Graphscited by 227
  3. 2019Contrastive Learning of Structured World Modelscited by 206
  4. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  5. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  6. 2020Keep CALM and Explore: Language Models for Action Generation in Text-based Gamescited by 108
  7. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  8. 2020Graph Constrained Reinforcement Learning for Natural Language Action Spacescited by 80
  9. 2015Language Understanding for Text-based Games Using Deep Reinforcement Learningcited by 70
  10. 2018Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learningcited by 53
  11. 2020Learning Dynamic Belief Graphs to Generalize on Text-Based Gamescited by 49
  12. 2015From Pixels to Torques: Policy Learning with Deep Dynamical Modelscited by 44
  13. 2020How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worldscited by 38
  14. 2020Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselinescited by 37
  15. 2020Playing Text-Based Games with Common Sensecited by 21
  16. 2019Learning Norms from Stories: A Prior for Value Aligned Agentscited by 19
  17. 2020Enhancing Text-based Reinforcement Learning Agents with Commonsense Knowledgecited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 7; 2024: 3; 2025: 6; 2026: 4.

31 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 249
  3. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  4. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  5. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  6. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
  7. 2022Large Language Models can Implement Policy Iterationcited by 31
  8. 2022I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragonscited by 22
  9. 2024Making Large Language Models into World Models with Precondition and Effect Knowledgecited by 21
  10. 2021A Systematic Survey of Text Worlds as Embodied Natural Language Environmentscited by 17
  11. 2021Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learningcited by 14
  12. 2021Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triplescited by 13
19 more
  1. 2022Behavior Cloned Transformers are Neurosymbolic Reasonerscited by 10
  2. 2025GraphThought: Graph Combinatorial Optimization with Thought Generationcited by 9
  3. 2025LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Studycited by 9
  4. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 5
  5. 2026Graph World Models: Concepts, Taxonomy, and Future Directionscited by 5
  6. 2025Graph-Grounded LLMs: Leveraging Graphical Function Calling to Minimize LLM Hallucinationscited by 4
  7. 2024DAVIS: Planning Agent with Knowledge Graph-Powered Inner Monologuecited by 3
  8. 2026Understanding Rollout Error in Graph World Modelscited by 3
  9. 2023Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Gamescited by 2
  10. 2022An Analysis of Deep Reinforcement Learning Agents for Text-based Gamescited by 1
  11. 2022Commonsense Knowledge from Scene Graphs for Textual Environmentscited by 1
  12. 2023Constructing Temporal Dynamic Knowledge Graphs from Interactive Text-based Gamescited by 1
  13. 2025KnowMap: Efficient Knowledge-Driven Task Adaptation for LLMscited by 1
  14. 2025Efficient Graph Understanding with LLMs via Structured Context Injectioncited by 1
  15. 2022DiffG-RL: Leveraging Difference between State and Common Sensecited by 0
  16. 2023Causal schema induction for knowledge discoverycited by 0
  17. 2024Can Language Models Enable In-Context Database?cited by 0
  18. 2026The Mechanism Matters: When Knowledge Graphs Help Reinforcement Learningcited by 0
  19. 2026HyperWorld: Hypergraph-Structured State Serialization Improves Learned Textual World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Modeling Worlds in Text
  2. 2023Knowledge-enhanced Agents for Interactive Text Games
  3. 2023Infusing Commonsense World Models with Graph Knowledge
  4. 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?
  5. 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agents
  6. 2025WorldLLM: Improving LLMs' world modeling using curiosity-driven theory-making
  7. 2025Text2World: Benchmarking Large Language Models for Symbolic World Model Generation
  8. 2025TextAtari: 100K Frames Game Playing with Language Agents