# Agentic Episodic Control

arXiv: https://arxiv.org/abs/2506.01442 · HTML: https://mlsanity.com/paper/2506.01442/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 37 of its 64 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 30 of 37)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Online Reinforcement Learning with Offline Data](https://mlsanity.com/paper/2302.02948/) (2023)
- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [ReFT: Reasoning with Reinforced Fine-Tuning](https://mlsanity.com/paper/2401.08967/) (2024)
- [MathPrompter: Mathematical Reasoning using Large Language Models](https://mlsanity.com/paper/2303.05398/) (2023)
- [Pre-Trained Language Models for Interactive Decision-Making](https://mlsanity.com/paper/2202.01771/) (2022)
- [Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning](https://mlsanity.com/paper/2302.02662/) (2023)
- [Embodied Intelligence via Learning and Evolution](https://mlsanity.com/paper/2102.02202/) (2021)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [Pretraining Representations for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2106.04799/) (2021)
- [Model-Free Episodic Control](https://mlsanity.com/paper/1606.04460/) (2016)
- [Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft](https://mlsanity.com/paper/2312.09238/) (2023)
- [Optimizing Agent Behavior over Long Time Scales by Transporting Value](https://mlsanity.com/paper/1810.06721/) (2018)
- [Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals](https://mlsanity.com/paper/2302.04449/) (2023)
- [Episodic Memory Deep Q-Networks](https://mlsanity.com/paper/1805.07603/) (2018)
- [Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2406.17169/) (2024)
- [Generalization of Reinforcement Learners with Working and Episodic Memory](https://mlsanity.com/paper/1910.13406/) (2019)
- [Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learning](https://mlsanity.com/paper/2308.14284/) (2023)
- [Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update](https://mlsanity.com/paper/1805.12375/) (2018)
- [LLM-Empowered State Representation for Reinforcement Learning](https://mlsanity.com/paper/2407.13237/) (2024)
- [Been There, Done That: Meta-Learning with Episodic Recall](https://mlsanity.com/paper/1805.09692/) (2018)
- [Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning](https://mlsanity.com/paper/2310.20587/) (2023)
- [Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models](https://mlsanity.com/paper/2312.17661/) (2023)
- [LLM-State: Open World State Representation for Long-horizon Task Planning with Large Language Model](https://mlsanity.com/paper/2311.17406/) (2023)

## Built on by (most cited first, 1 of 1)

- [PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agents](https://mlsanity.com/paper/2605.08468/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) remains fundamentally limited by poor data efficiency and weak generalization. Prior episodic RL methods attempt to alleviate this via external memory modules, yet they suffer from two key limitations: a representation bottleneck caused by shallow encoders, and a retrieval dilemma where episodic memory is accessed indiscriminately. To address these challenges, we propose Agentic Episodic Control (AEC), a novel architecture that integrates large language models (LLMs) into episodic RL. AEC uses an LLM-based semantic augmenter to generate semantic representations from raw observations, and a critical state recognizer to selectively retrieve valuable experiences. This transforms memory usage from passive similarity matching into strategic, context-aware recall. Across five BabyAI-Text environments, AEC achieves 2-6x higher data efficiency than baselines and is the only method to solve complex tasks like UnlockLocal with over 90% success. It further demonstrates strong cross-task and cross-environment generalization, maintaining performance even under distribution shifts. AEC shows that combining LLM-derived priors with reinforcement learning yields more sample-efficient and adaptable agents.