mlsanityData to 7 Oct 2026

Towards Improving Sequential Decision-Making in LLM Agents via Experience Memory

arXiv 2608.03420 ↗First posted 4 Aug 2026cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large language models have improved substantially on single-shot reasoning tasks, but their performance in sequential decision-making is less well understood. We study this on fully-observable two-player zero-sum games, which provide ground-truth evaluation: outcomes are determined by the rules, and optimality of individual moves can be computed or approximated, without relying on a judge model. Across model tiers, LLMs play suboptimally in simple games such as tic-tac-toe or Connect Four, and lose to MCTS opponents. Obfuscations that preserve the game tree but rewrite its surface form leave performance largely unchanged, indicating the gap is not fully explained by recall of memorized strategies. Motivated by this performance gap, we introduce an agentic framework enhanced with an experience memory designed for the sequential setting and addressing common challenges of sequential decision-making such as credit assignment. We show that post-game reflection and rule extraction yield measurable improvements on tic-tac-toe without modifying the model weights.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 13 of its 17 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

13 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 12

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  7. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  8. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  9. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  10. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 119
  11. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  12. 2025Code World Models for General Game Playingcited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents
  2. 2026MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games
  3. 2026PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning
  4. 2026AdMem: Advanced Memory for Task-solving Agents
  5. 2025From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory
  6. 2026LLMs Are Not Good Strategists, Yet Memory-Enhanced Agency Boosts Reasoning
  7. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Games
  8. 2026From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory