# Neural Episodic Control with State Abstraction

arXiv: https://arxiv.org/abs/2301.11490 · HTML: https://mlsanity.com/paper/2301.11490/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 65 references that are in the map.
- First posted on arXiv 27 Jan 2023.

## Builds on (most cited first, 30 of 32)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Neural Episodic Control](https://mlsanity.com/paper/1703.01988/) (2017)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [Model-Free Episodic Control](https://mlsanity.com/paper/1606.04460/) (2016)
- [Discretizing Continuous Action Space for On-Policy Optimization](https://mlsanity.com/paper/1901.10500/) (2019)
- [Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration](https://mlsanity.com/paper/2111.11032/) (2021)
- [Episodic Memory Deep Q-Networks](https://mlsanity.com/paper/1805.07603/) (2018)
- [Offline Reinforcement Learning with Value-based Episodic Memory](https://mlsanity.com/paper/2110.09796/) (2021)
- [Learning Guidance Rewards with Trajectory-space Smoothing](https://mlsanity.com/paper/2010.12718/) (2020)
- [Experiments with Random Projection](https://mlsanity.com/paper/1301.3849/) (2013)
- [Generalizable Episodic Memory for Deep Reinforcement Learning](https://mlsanity.com/paper/2103.06469/) (2021)
- [Automatic Web Testing using Curiosity-Driven Reinforcement Learning](https://mlsanity.com/paper/2103.06018/) (2021)
- [GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesis](https://mlsanity.com/paper/2205.13728/) (2022)
- [Fast deep reinforcement learning using online adjustments from the past](https://mlsanity.com/paper/1810.08163/) (2018)
- [Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments](https://mlsanity.com/paper/1903.00827/) (2019)
- [Model-Based Episodic Memory Induces Dynamic Hybrid Controls](https://mlsanity.com/paper/2111.02104/) (2021)
- [KoGuN: Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge](https://mlsanity.com/paper/2002.07418/) (2020)
- [Solving Continuous Control with Episodic Memory](https://mlsanity.com/paper/2106.08832/) (2021)
- [Guided Policy Search Model-based Reinforcement Learning for Urban Autonomous Driving](https://mlsanity.com/paper/2005.03076/) (2020)
- [Episodic Policy Gradient Training](https://mlsanity.com/paper/2112.01853/) (2021)

## Built on by (most cited first, 7 of 7)

- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Episodic Reinforcement Learning with Expanded State-reward Space](https://mlsanity.com/paper/2401.10516/) (2024)
- [Trajectory-Class-Aware Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.01440/) (2025)
- [Subgoal Discovery Using a Free Energy Paradigm and State Aggregations](https://mlsanity.com/paper/2412.16687/) (2024)
- [Agentic Episodic Control](https://mlsanity.com/paper/2506.01442/) (2025)
- [Robustness Verification of Deep Reinforcement Learning Based Control Systems using Reward Martingales](https://mlsanity.com/paper/2312.09695/) (2023)
- [Memory-Driven Self-Improvement for Decision Making with Large Language Models](https://mlsanity.com/paper/2509.26340/) (2025)

## Abstract (arXiv)

Existing Deep Reinforcement Learning (DRL) algorithms suffer from sample inefficiency. Generally, episodic control-based approaches are solutions that leverage highly-rewarded past experiences to improve sample efficiency of DRL algorithms. However, previous episodic control-based approaches fail to utilize the latent information from the historical behaviors (e.g., state transitions, topological similarities, etc.) and lack scalability during DRL training. This work introduces Neural Episodic Control with State Abstraction (NECSA), a simple but effective state abstraction-based episodic control containing a more comprehensive episodic memory, a novel state evaluation, and a multi-step state analysis. We evaluate our approach to the MuJoCo and Atari tasks in OpenAI gym domains. The experimental results indicate that NECSA achieves higher sample efficiency than the state-of-the-art episodic control-based approaches. Our data and code are available at the project websitefootnoteurlhttps://sites.google.com/view/drl-necsa.