mlsanityData to 7 Oct 2026

HERO'S JOURNEY: Testing Complex Rule Induction with Text Games

arXiv 2606.02556 ↗First posted 1 Jun 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce HERO'S JOURNEY, a benchmark for rule induction in goal-directed episodic tasks, where agents must infer hidden rules from demonstrations and act on them through multi-step execution. HERO'S JOURNEY covers eight tasks across attribute and procedural induction families, each with four structural rule forms, controllable lexical grounding, and identifiability conditions. Evaluating state-of-the-art LLMs, we find that models show evidence of rule induction, but the ability is limited and uneven across tasks. Meanwhile, process execution adds an execution bottleneck for models, whereas surface semantics has minimal effect. Induction-specific steering methods improve performance on attribute tasks but show no reliable gains on procedural tasks, suggesting the gap in procedural induction remains an open challenge.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

18 of 30 references.

References · 18

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  6. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  7. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  8. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  9. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  10. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  11. 2026Towards End-to-End Automation of AI Researchcited by 150
  12. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
6 more
  1. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  2. 2021ACRE: Abstract Causal REasoning Beyond Covariationcited by 30
  3. 2024IDEA: Enhancing the Rule Learning Ability of Large Language Model Agent through Induction, Deduction, and Abductioncited by 20
  4. 2024MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Modelscited by 17
  5. 2024Generating Diverse Hypotheses for Inductive Reasoningcited by 6
  6. 2025On Language Models' Sensitivity to Suspicious Coincidencescited by 2

Built on by

Not cited yet.