# CONTHER: Context-Aware Reinforcement Learning for Robotic Manipulation with Sparse Rewards

arXiv: https://arxiv.org/abs/2503.15895 · HTML: https://mlsanity.com/paper/2503.15895/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 17 of its 19 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [On Improving Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1804.06309/) (2018)

## Abstract (arXiv)

This paper investigates whether sequential context improves goal-conditioned Reinforcement Learning in sparse-reward manipulation tasks. While Hindsight Experience Replay (HER) addresses reward sparsity through goal relabeling, its operation on isolated transitions limits its ability to capture temporal dependencies inherent in joint-space control. We hypothesize that incorporating motion history can enhance policy learning and introduce CONTHER, which integrates a Transformer-based architecture with a modified HER replay buffer. The Transformer encodes sequences of prior states and goals to provide temporal awareness, while the buffer populates experience with artificially successful trajectories. Two architectural variants are analyzed to examine how contextual information should be integrated. In simulated point-reaching tasks with a UR3 manipulator, CONTHER achieves a 38.46% higher average success rate compared to baselines and outperforms the strongest baseline by 28.21%, with faster convergence and more stable learning. The framework is further evaluated on three dynamic tasks requiring complex trajectory following and obstacle avoidance, where temporal context is critical. By operating directly on joint velocities, the approach provides a foundation for transfer to physical systems. The primary contribution is a systematic investigation into fusing sequential context with goal relabeling, offering insights into how temporal awareness benefits policy learning.