# Contact Energy Based Hindsight Experience Prioritization

arXiv: https://arxiv.org/abs/2312.02677 · HTML: https://mlsanity.com/paper/2312.02677/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 17 of 17)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger](https://mlsanity.com/paper/1803.00628/) (2018)
- [Maximum Entropy-Regularized Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/1905.08786/) (2019)
- [SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation](https://mlsanity.com/paper/2101.11812/) (2021)
- [Energy-Based Hindsight Experience Prioritization](https://mlsanity.com/paper/1810.01363/) (2018)
- [Meta-Reinforcement Learning via Language Instructions](https://mlsanity.com/paper/2209.04924/) (2022)
- [Leveraging Contact Forces for Learning to Grasp](https://mlsanity.com/paper/1809.07004/) (2018)
- [Improved Learning of Robot Manipulation Tasks via Tactile Intrinsic Motivation](https://mlsanity.com/paper/2102.11051/) (2021)
- [Meta-Reinforcement Learning in Broad and Non-Parametric Environments](https://mlsanity.com/paper/2108.03718/) (2021)
- [Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning](https://mlsanity.com/paper/2305.00286/) (2023)
- [Learning from Symmetry: Meta-Reinforcement Learning with Symmetrical Behaviors and Language Instructions](https://mlsanity.com/paper/2209.10656/) (2022)
- [Complex Robotic Manipulation via Graph-Based Hindsight Goal Generation](https://mlsanity.com/paper/2007.13486/) (2020)

## Built on by (most cited first, 2 of 2)

- [Efficient Diversity-based Experience Replay for Deep Reinforcement Learning](https://mlsanity.com/paper/2410.20487/) (2024)
- [Fuzzy Logic Theory-based Adaptive Reward Shaping for Robust Reinforcement Learning (FARS)](https://mlsanity.com/paper/2604.15772/) (2026)

## Abstract (arXiv)

Multi-goal robot manipulation tasks with sparse rewards are difficult for reinforcement learning (RL) algorithms due to the inefficiency in collecting successful experiences. Recent algorithms such as Hindsight Experience Replay (HER) expedite learning by taking advantage of failed trajectories and replacing the desired goal with one of the achieved states so that any failed trajectory can be utilized as a contribution to learning. However, HER uniformly chooses failed trajectories, without taking into account which ones might be the most valuable for learning. In this paper, we address this problem and propose a novel approach Contact Energy Based Prioritization~(CEBP) to select the samples from the replay buffer based on rich information due to contact, leveraging the touch sensors in the gripper of the robot and object displacement. Our prioritization scheme favors sampling of contact-rich experiences, which are arguably the ones providing the largest amount of information. We evaluate our proposed approach on various sparse reward robotic tasks and compare them with the state-of-the-art methods. We show that our method surpasses or performs on par with those methods on robot manipulation tasks. Finally, we deploy the trained policy from our method to a real Franka robot for a pick-and-place task. We observe that the robot can solve the task successfully. The videos and code are publicly available at: https://erdiphd.github.io/HER_force