# There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning

arXiv: https://arxiv.org/abs/2106.04480 · HTML: https://mlsanity.com/paper/2106.04480/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 65 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [Acme: A Research Framework for Distributed Reinforcement Learning](https://mlsanity.com/paper/2006.00979/) (2020)
- [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [AI Safety Gridworlds](https://mlsanity.com/paper/1711.09883/) (2017)
- [Episodic Curiosity through Reachability](https://mlsanity.com/paper/1810.02274/) (2018)
- [Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning](https://mlsanity.com/paper/2004.14646/) (2020)
- [Playing hard exploration games by watching YouTube](https://mlsanity.com/paper/1805.11592/) (2018)
- [Munchausen Reinforcement Learning](https://mlsanity.com/paper/2007.14430/) (2020)
- [An investigation of model-free planning](https://mlsanity.com/paper/1901.03559/) (2019)
- [Unsupervised Learning of Spatiotemporally Coherent Metrics](https://mlsanity.com/paper/1412.6056/) (2014)
- [Neural Predictive Belief Representations](https://mlsanity.com/paper/1811.06407/) (2018)
- [Safe Learning of Regions of Attraction for Uncertain, Nonlinear Systems with Gaussian Processes](https://mlsanity.com/paper/1603.04915/) (2016)
- [Safe Exploration of State and Action Spaces in Reinforcement Learning](https://mlsanity.com/paper/1402.0560/) (2014)
- [Learning Actionable Representations from Visual Observations](https://mlsanity.com/paper/1808.00928/) (2018)
- [Can Temporal Information Help with Contrastive Self-Supervised Learning?](https://mlsanity.com/paper/2011.13046/) (2020)
- [Learning from Heterogeneous EEG Signals with Differentiable Channel Reordering](https://mlsanity.com/paper/2010.13694/) (2020)
- [Self-Supervised Learning of Audio Representations from Permutations with Differentiable Ranking](https://mlsanity.com/paper/2103.09879/) (2021)
- [Motion Perception in Reinforcement Learning with Dynamic Objects](https://mlsanity.com/paper/1901.03162/) (2019)
- [Skip-Clip: Self-Supervised Spatiotemporal Representation Learning by Future Clip Order Ranking](https://mlsanity.com/paper/1910.12770/) (2019)
- [Learning Temporal Embeddings for Complex Video Analysis](https://mlsanity.com/paper/1505.00315/) (2015)

## Built on by (most cited first, 18 of 18)

- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning](https://mlsanity.com/paper/2210.10765/) (2022)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Automating Reinforcement Learning with Example-based Resets](https://mlsanity.com/paper/2204.02041/) (2022)
- [Robust Imitation of a Few Demonstrations with a Backwards Model](https://mlsanity.com/paper/2210.09337/) (2022)
- [Avoiding Catastrophe in Online Learning by Asking for Help](https://mlsanity.com/paper/2402.08062/) (2024)
- [PushWorld: A benchmark for manipulation planning with tools and movable obstacles](https://mlsanity.com/paper/2301.10289/) (2023)
- [When Learning Is Out of Reach, Reset: Generalization in Autonomous Visuomotor Reinforcement Learning](https://mlsanity.com/paper/2303.17600/) (2023)
- [Learning to Undo: Rollback-Augmented Reinforcement Learning with Reversibility Signals](https://mlsanity.com/paper/2510.14503/) (2025)
- [An Investigation of Time Reversal Symmetry in Reinforcement Learning](https://mlsanity.com/paper/2311.17008/) (2023)
- [More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences](https://mlsanity.com/paper/2110.10632/) (2021)
- [Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learning](https://mlsanity.com/paper/2505.13925/) (2025)
- [Approximating Safety Feedback Without a Safety Oracle via Model Predictive Control](https://mlsanity.com/paper/2510.20955/) (2025)
- [Inverse Manipulation through Symbolic Planning and Residual Operator Learning](https://mlsanity.com/paper/2606.05248/) (2026)
- [REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff](https://mlsanity.com/paper/2609.17745/) (2026)
- [Beyond Patch Removal: Persistent Adversarial Effects in Vision-Language-Action Policies](https://mlsanity.com/paper/2609.19669/) (2026)
- [Nociception as a Control Primitive: Afferent Channels and Nociceptive Memory for Agents Deployed in One Body](https://mlsanity.com/paper/2609.34840/) (2026)
- [DeepRewind: Predicting and Repairing Premature Commitments in Deep Research Agents](https://mlsanity.com/paper/2609.36344/) (2026)

## Abstract (arXiv)

We propose to learn to distinguish reversible from irreversible actions for better informed decision-making in Reinforcement Learning (RL). From theoretical considerations, we show that approximate reversibility can be learned through a simple surrogate task: ranking randomly sampled trajectory events in chronological order. Intuitively, pairs of events that are always observed in the same order are likely to be separated by an irreversible sequence of actions. Conveniently, learning the temporal order of events can be done in a fully self-supervised way, which we use to estimate the reversibility of actions from experience, without any priors. We propose two different strategies that incorporate reversibility in RL agents, one strategy for exploration (RAE) and one strategy for control (RAC). We demonstrate the potential of reversibility-aware agents in several environments, including the challenging Sokoban game. In synthetic tasks, we show that we can learn control policies that never fail and reduce to zero the side-effects of interactions, even without access to the reward function.