mlsanityData to 7 Oct 2026

CAST: Game Solvers as Turn-Level Teachers for LLM Agents

arXiv 2607.25308 ↗First posted 28 Jul 2026cs.CL cs.AIcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at https://github.com/Wloner0809/CAST.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 43 of its 54 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

43 of 54 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Look Before You Leap: Autonomous Exploration for LLM Agents15 May
  2. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents8 Apr
  3. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  4. 2026ProAct: Agentic Lookahead in Interactive Environments5 Feb
  5. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs18 Dec
  6. 2025AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress11 Nov
  7. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents19 Oct
  8. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning30 Jun
  9. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution27 May
  10. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  11. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games20 Nov
  12. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
2 more
  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun

Other references · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
17 more
  1. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  3. 2025Group Sequence Policy Optimizationcited by 788
  4. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  5. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  6. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  7. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  8. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  9. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  10. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  11. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 53
  12. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  13. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 32
  14. 2025G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learningcited by 29
  15. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 18
  16. 2025Play to Generalize: Learning to Reason Through Game Playcited by 15
  17. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Usecited by 0
  2. 2026Consistent Plan-Act for Long-Horizon Agentic Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization
  2. 2026Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks
  3. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
  4. 2024Large Language Models as Agents in Two-Player Games
  5. 2025Learning Game-Playing Agents with Generative Code Optimization
  6. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
  7. 2026VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
  8. 2026Towards Improving Sequential Decision-Making in LLM Agents via Experience Memory