# Energy-Based Hindsight Experience Prioritization

arXiv: https://arxiv.org/abs/1810.01363 · HTML: https://mlsanity.com/paper/1810.01363/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2018.

## Built on by (most cited first, 21 of 21)

- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [Causal Influence Detection for Improving Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2106.03443/) (2021)
- [Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewards](https://mlsanity.com/paper/2208.00843/) (2022)
- [Episodic Self-Imitation Learning with Hindsight](https://mlsanity.com/paper/2011.13467/) (2020)
- [Improved Learning of Robot Manipulation Tasks via Tactile Intrinsic Motivation](https://mlsanity.com/paper/2102.11051/) (2021)
- [PTR-PPO: Proximal Policy Optimization with Prioritized Trajectory Replay](https://mlsanity.com/paper/2112.03798/) (2021)
- [Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following](https://mlsanity.com/paper/2512.23457/) (2025)
- [Diversity-based Trajectory and Goal Selection with Hindsight Experience Replay](https://mlsanity.com/paper/2108.07887/) (2021)
- [Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment](https://mlsanity.com/paper/2110.15043/) (2021)
- [Efficient Diversity-based Experience Replay for Deep Reinforcement Learning](https://mlsanity.com/paper/2410.20487/) (2024)
- [Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning](https://mlsanity.com/paper/2102.12962/) (2021)
- [Hindsight Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1903.07854/) (2019)
- [SVL: Goal-Conditioned Reinforcement Learning as Survival Learning](https://mlsanity.com/paper/2604.17551/) (2026)
- [Hindsight States: Blending Sim and Real Task Elements for Efficient Reinforcement Learning](https://mlsanity.com/paper/2303.02234/) (2023)
- [Contact Energy Based Hindsight Experience Prioritization](https://mlsanity.com/paper/2312.02677/) (2023)
- [Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization](https://mlsanity.com/paper/2312.05787/) (2023)
- [ACDC: Adaptive Curriculum Planning with Dynamic Contrastive Control for Goal-Conditioned Reinforcement Learning in Robotic Manipulation](https://mlsanity.com/paper/2603.02104/) (2026)
- [Cluster-based Sampling in Hindsight Experience Replay for Robotic Tasks (Student Abstract)](https://mlsanity.com/paper/2208.14741/) (2022)
- [Next-Future: Sample-Efficient Policy Learning for Robotic-Arm Tasks](https://mlsanity.com/paper/2504.11247/) (2025)
- [Path-level Hindsight Instructions for Semantic Exploration in Vision-Language Navigation](https://mlsanity.com/paper/2607.01754/) (2026)
- [Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF](https://mlsanity.com/paper/2607.07693/) (2026)

## Abstract (arXiv)

In Hindsight Experience Replay (HER), a reinforcement learning agent is trained by treating whatever it has achieved as virtual goals. However, in previous work, the experience was replayed at random, without considering which episode might be the most valuable for learning. In this paper, we develop an energy-based framework for prioritizing hindsight experience in robotic manipulation tasks. Our approach is inspired by the work-energy principle in physics. We define a trajectory energy function as the sum of the transition energy of the target object over the trajectory. We hypothesize that replaying episodes that have high trajectory energy is more effective for reinforcement learning in robotics. To verify our hypothesis, we designed a framework for hindsight experience prioritization based on the trajectory energy of goal states. The trajectory energy function takes the potential, kinetic, and rotational energy into consideration. We evaluate our Energy-Based Prioritization (EBP) approach on four challenging robotic manipulation tasks in simulation. Our empirical results show that our proposed method surpasses state-of-the-art approaches in terms of both performance and sample-efficiency on all four tasks, without increasing computational time. A video showing experimental results is available at https://youtu.be/jtsF2tTeUGQ