# PALMER: Perception-Action Loop with Memory for Long-Horizon Planning

arXiv: https://arxiv.org/abs/2212.04581 · HTML: https://mlsanity.com/paper/2212.04581/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 69 references that are in the map.
- First posted on arXiv 8 Dec 2022.

## Builds on (most cited first, 23 of 23)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Learning to Poke by Poking: Experiential Learning of Intuitive Physics](https://mlsanity.com/paper/1606.07419/) (2016)
- [Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills](https://mlsanity.com/paper/2104.07749/) (2021)
- [Rapid Exploration for Open-World Navigation with Latent Goal Models](https://mlsanity.com/paper/2104.05859/) (2021)
- [Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery](https://mlsanity.com/paper/1907.08225/) (2019)
- [ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints](https://mlsanity.com/paper/2202.11271/) (2022)
- [Model-Based Visual Planning with Self-Supervised Functional Distances](https://mlsanity.com/paper/2012.15373/) (2020)
- [Scaling Local Control to Large-Scale Topological Navigation](https://mlsanity.com/paper/1909.12329/) (2019)
- [Sparse Graphical Memory for Robust Planning](https://mlsanity.com/paper/2003.06417/) (2020)
- [Ecological Reinforcement Learning](https://mlsanity.com/paper/2006.12478/) (2020)

## Built on by (most cited first, 5 of 5)

- [Horizon Generalization in Reinforcement Learning](https://mlsanity.com/paper/2501.02709/) (2025)
- [Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2302.10639/) (2023)
- [Compositional Servoing by Recombining Demonstrations](https://mlsanity.com/paper/2310.04271/) (2023)
- [Aim Short to Reach Far: Your Frozen World Model Can Plan Better Than You Think](https://mlsanity.com/paper/2609.30036/) (2026)
- [Ultrasound Operator Guidance Using World Modeling and Retrieval Based Action Planning](https://mlsanity.com/paper/2610.06008/) (2026)

## Abstract (arXiv)

To achieve autonomy in a priori unknown real-world scenarios, agents should be able to: i) act from high-dimensional sensory observations (e.g., images), ii) learn from past experience to adapt and improve, and iii) be capable of long horizon planning. Classical planning algorithms (e.g. PRM, RRT) are proficient at handling long-horizon planning. Deep learning based methods in turn can provide the necessary representations to address the others, by modeling statistical contingencies between observations. In this direction, we introduce a general-purpose planning algorithm called PALMER that combines classical sampling-based planning algorithms with learning-based perceptual representations. For training these perceptual representations, we combine Q-learning with contrastive representation learning to create a latent space where the distance between the embeddings of two states captures how easily an optimal policy can traverse between them. For planning with these perceptual representations, we re-purpose classical sampling-based planning algorithms to retrieve previously observed trajectory segments from a replay buffer and restitch them into approximately optimal paths that connect any given pair of start and goal states. This creates a tight feedback loop between representation learning, memory, reinforcement learning, and sampling-based planning. The end result is an experiential framework for long-horizon planning that is significantly more robust and sample efficient compared to existing methods.