# Memory Allocation in Resource-Constrained Reinforcement Learning

arXiv: https://arxiv.org/abs/2506.17263 · HTML: https://mlsanity.com/paper/2506.17263/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 11 of 11)

- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023)
- [People construct simplified mental representations to plan](https://mlsanity.com/paper/2105.06948/) (2021)
- [Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent States](https://mlsanity.com/paper/2102.05261/) (2021)
- [Continual Learning as Computationally Constrained Reinforcement Learning](https://mlsanity.com/paper/2307.04345/) (2023)
- [Selective Dyna-style Planning Under Limited Model Capacity](https://mlsanity.com/paper/2007.02418/) (2020)
- [Jelly Bean World: A Testbed for Never-Ending Learning](https://mlsanity.com/paper/2002.06306/) (2020)
- [Prediction and Control in Continual Reinforcement Learning](https://mlsanity.com/paper/2312.11669/) (2023)
- [The Need for a Big World Simulator: A Scientific Challenge for Continual Learning](https://mlsanity.com/paper/2408.02930/) (2024)
- [Satisficing Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/2407.12185/) (2024)

## Built on by (most cited first, 2 of 2)

- [Position: Deployed Reinforcement Learning should be Continual](https://mlsanity.com/paper/2606.04029/) (2026)
- [Artifacts as Memory Beyond the Agent Boundary](https://mlsanity.com/paper/2604.08756/) (2026)

## Abstract (arXiv)

Resource constraints can fundamentally change both learning and decision-making. We explore how memory constraints influence an agent's performance when navigating unknown environments using standard reinforcement learning algorithms. Specifically, memory-constrained agents face a dilemma: how much of their limited memory should be allocated to each of the agent's internal processes, such as estimating a world model, as opposed to forming a plan using that model? We study this dilemma in MCTS- and DQN-based algorithms and examine how different allocations of memory impact performance in episodic and continual learning settings.