mlsanityData to 14 Oct 2026

ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games

arXiv 2305.14879 ↗First posted 24 May 2023cs.CL cs.AIcited by 23

On the map: World models → World models for language agents

Abstract (arXiv)

In this work, we investigate the capacity of language models to generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks. We operationalize this as a task of generating text games, expressed as hundreds of lines of Python code. To facilitate this task, we introduce ByteSized32 (Code: github.com/cognitiveailab/BYTESIZED32), a corpus of 32 reasoning-focused text games totaling 20k lines of Python code. We empirically demonstrate that GPT-4 can use these games as templates for single-shot in-context learning, successfully producing runnable games on unseen topics in 28% of cases. When allowed to self-reflect on program errors, game runnability substantially increases to 57%. While evaluating simulation fidelity is labor-intensive, we introduce a suite of automated metrics to assess game fidelity, technical validity, adherence to task specifications, and winnability, showing a high degree of agreement with expert human ratings. We pose this as a challenge task to spur further development at the juncture of world modeling and code generation.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 27 of its 45 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

27 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learning8 Oct

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2016OpenAI Gymcited by 1,736
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  9. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  10. 2023Teaching Large Language Models to Self-Debugcited by 988
  11. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  12. 2023StarCoder: may the source be with you!cited by 799
14 more
  1. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  2. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  3. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  4. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  5. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  6. 2019Interactive Fiction Games: A Colossal Adventurecited by 168
  7. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  8. 2019Learning to Speak and Act in a Fantasy Text Adventure Gamecited by 122
  9. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 98
  10. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  11. 2023Conversational Automated Program Repaircited by 41
  12. 2023ChatGPT: Applications, Opportunities, and Threatscited by 28
  13. 2021A Systematic Survey of Text Worlds as Embodied Natural Language Environmentscited by 17
  14. 2019Generating Interactive Worlds with Textcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 11; 2026: 2.

23 papers.

Sort:

By area:

  1. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 83
  2. 2024Can Language Models Serve as Text-Based World Simulators?cited by 52
  3. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 46
  4. 2025Text2World: Benchmarking Large Language Models for Symbolic World Model Generationcited by 26
  5. 2024A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debatescited by 23
  6. 2025WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planningcited by 20
  7. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  8. 2024Unbounded: A Generative Infinite Game of Character Life Simulationcited by 14
  9. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  10. 2024LLM-Based World Models Can Make Decisions Solely, But Rigorous Evaluations are Neededcited by 10
  11. 2024Missed Connections: Lateral Thinking Puzzles for Large Language Modelscited by 9
  12. 2024LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planningcited by 9
11 more
  1. 2025Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedbackcited by 9
  2. 2024STARLING: Self-supervised Training of Text-based Reinforcement Learning Agent with Large Language Modelscited by 8
  3. 2025PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilitiescited by 7
  4. 2025A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembenchcited by 3
  5. 2025Language Model as Planner and Formalizer under Constraintscited by 3
  6. 2026AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agentscited by 3
  7. 2024Open Role-Playing with Delta-Enginescited by 1
  8. 2025Iterative Formalization and Planning in Partially Observable Environmentscited by 1
  9. 2025ByteSized32Refactored: Towards an Extensible Interactive Text Games Corpus for LLM World Modeling and Evaluationcited by 0
  10. 2025Telling Speculative Stories to Help Humans Imagine the Harms of Healthcare AIcited by 0
  11. 2026World Editing: Intervening on Executable Worlds at Increasing Depthcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Distilling Game Code World Model Generation into Lightweight Large Language Models
  2. 2025TextGames: Learning to Self-Play Text-Based Puzzle Games via Language Model Reasoning
  3. 2021Modeling Worlds in Text
  4. 2026The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
  5. 2025Code World Models for General Game Playing
  6. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  7. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  8. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games