mlsanityData to 14 Oct 2026

Agentic Episodic Control

arXiv 2506.01442 ↗First posted 2 Jun 2025cs.AIcited by 1

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Reinforcement learning (RL) remains fundamentally limited by poor data efficiency and weak generalization. Prior episodic RL methods attempt to alleviate this via external memory modules, yet they suffer from two key limitations: a representation bottleneck caused by shallow encoders, and a retrieval dilemma where episodic memory is accessed indiscriminately. To address these challenges, we propose Agentic Episodic Control (AEC), a novel architecture that integrates large language models (LLMs) into episodic RL. AEC uses an LLM-based semantic augmenter to generate semantic representations from raw observations, and a critical state recognizer to selectively retrieve valuable experiences. This transforms memory usage from passive similarity matching into strategic, context-aware recall. Across five BabyAI-Text environments, AEC achieves 2-6x higher data efficiency than baselines and is the only method to solve complex tasks like UnlockLocal with over 90% success. It further demonstrates strong cross-task and cross-environment generalization, maintaining performance even under distribution shifts. AEC shows that combining LLM-derived priors with reinforcement learning yields more sample-efficient and adaptable agents.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 37 of its 64 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

37 of 64 references.

References · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  6. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  7. 2023Reward Design with Language Modelscited by 261
  8. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  9. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  10. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  11. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  12. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
25 more
  1. 2021Embodied Intelligence via Learning and Evolutioncited by 163
  2. 2017Neural Episodic Controlcited by 151
  3. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  4. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  5. 2016Model-Free Episodic Controlcited by 102
  6. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 61
  7. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 58
  8. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  9. 2018Episodic Memory Deep Q-Networkscited by 42
  10. 2024Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Modelscited by 39
  11. 2019Generalization of Reinforcement Learners with Working and Episodic Memorycited by 36
  12. 2023Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learningcited by 36
  13. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
  14. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  15. 2018Been There, Done That: Meta-Learning with Episodic Recallcited by 26
  16. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  17. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 16
  18. 2023LLM-State: Open World State Representation for Long-horizon Task Planning with Large Language Modelcited by 15
  19. 2018Fast deep reinforcement learning using online adjustments from the pastcited by 14
  20. 2023LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agentscited by 11
  21. 2023LANCAR: Leveraging Language for Context-Aware Robot Locomotion in Unstructured Environmentscited by 9
  22. 2023Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Modelscited by 9
  23. 2024A Survey Analyzing Generalization in Deep Reinforcement Learningcited by 8
  24. 2023Neural Episodic Control with State Abstractioncited by 7
  25. 2024Episodic Reinforcement Learning with Expanded State-reward Spacecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents
  2. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents
  3. 2026AdMem: Advanced Memory for Task-solving Agents
  4. 2026AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizing
  5. 2026Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents
  6. 2026Verifiable Memory: Learning Unified Memory Management with Local and Global Verifiers for Large Language Model Agents
  7. 2025Mem-α: Learning Memory Construction via Reinforcement Learning
  8. 2026The AI Hippocampus: How Far are We From Human Memory?