# The Effect of State Representation on LLM Agent Behavior in Dynamic Routing Games

arXiv: https://arxiv.org/abs/2506.15624 · HTML: https://mlsanity.com/paper/2506.15624/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 47 references that are in the map.
- First posted on arXiv 18 Jun 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 15 of 15)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection](https://mlsanity.com/paper/2306.04637/) (2023)
- [LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay](https://mlsanity.com/paper/2310.14985/) (2023)
- [LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory](https://mlsanity.com/paper/2408.02784/) (2024)
- [Do LLM Agents Have Regret? A Case Study in Online Learning and Games](https://mlsanity.com/paper/2403.16843/) (2024)
- [TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs](https://mlsanity.com/paper/2410.10479/) (2024)
- [Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Information](https://mlsanity.com/paper/2408.02559/) (2024)
- [GLEE: A Unified Framework and Benchmark for Language-based Economic Environments](https://mlsanity.com/paper/2410.05254/) (2024)
- [LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory](https://mlsanity.com/paper/2502.20432/) (2025)
- [LLM-driven Imitation of Subrational Behavior : Illusion or Reality?](https://mlsanity.com/paper/2402.08755/) (2024)
- [Can AI with High Reasoning Ability Replicate Human-like Decision Making in Economic Experiments?](https://mlsanity.com/paper/2406.11426/) (2024)

## Built on by (most cited first, 9 of 9)

- [Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning](https://mlsanity.com/paper/2508.19828/) (2025)
- [Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents](https://mlsanity.com/paper/2601.01885/) (2026)
- [A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents](https://mlsanity.com/paper/2506.23844/) (2025)
- [Exploring Silicon-Based Societies: An Early Study of the Moltbook Agent Community](https://mlsanity.com/paper/2602.02613/) (2026)
- [Simulating and Experimenting with Social Media Mobilization Using LLM Agents](https://mlsanity.com/paper/2510.26494/) (2025)
- [Competition and Cooperation of LLM Agents in Games](https://mlsanity.com/paper/2604.00487/) (2026)
- [Verifiable Memory: Learning Unified Memory Management with Local and Global Verifiers for Large Language Model Agents](https://mlsanity.com/paper/2608.03137/) (2026)
- [State Design Matters: How Representations Shape Dynamic Reasoning in Large Language Models](https://mlsanity.com/paper/2602.15858/) (2026)
- [Warned alike, AI agents avoid the less-crowded road while people take it](https://mlsanity.com/paper/2609.30883/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown promise as decision-makers in dynamic settings, but their stateless nature necessitates creating a natural language representation of history. We present a unifying framework for systematically constructing natural language "state" representations for prompting LLM agents in repeated multi-agent games. Previous work on games with LLM agents has taken an ad hoc approach to encoding game history, which not only obscures the impact of state representation on agents' behavior, but also limits comparability between studies. Our framework addresses these gaps by characterizing methods of state representation along three axes: action informativeness (i.e., the extent to which the state representation captures actions played); reward informativeness (i.e., the extent to which the state representation describes rewards obtained); and prompting style (or natural language compression, i.e., the extent to which the full text history is summarized). We apply this framework to a dynamic selfish routing game, chosen because it admits a simple equilibrium both in theory and in human subject experiments citerapoportchoice2009. Despite the game's relative simplicity, we find that there are key dependencies of LLM agent behavior on the natural language state representation. In particular, we observe that representations which provide agents with (1) summarized, rather than complete, natural language representations of past history; (2) information about regrets, rather than raw payoffs; and (3) limited information about others' actions lead to behavior that more closely matches game theoretic equilibrium predictions, and with more stable game play by the agents. By contrast, other representations can exhibit either large deviations from equilibrium, higher variation in dynamic game play over time, or both.