# The Geometry of Learning to Avoid Interventions

arXiv: https://arxiv.org/abs/2602.03825 · HTML: https://mlsanity.com/paper/2602.03825/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 10 of 10)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [Bridging the Gap Between Value and Policy Based Reinforcement Learning](https://mlsanity.com/paper/1702.08892/) (2017)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [RLIF: Interactive Imitation Learning as Reinforcement Learning](https://mlsanity.com/paper/2311.12996/) (2023)
- [Residual Q-Learning: Offline and Online Policy Customization without Value](https://mlsanity.com/paper/2306.09526/) (2023)
- [MILE: Model-based Intervention Learning](https://mlsanity.com/paper/2502.13519/) (2025)
- [Mo' States Mo' Problems: Emergency Stop Mechanisms from Observation](https://mlsanity.com/paper/1912.01649/) (2019)

## Abstract (arXiv)

Human interventions are a common source of supervision in autonomous systems during deployment. Many existing approaches are based on avoiding interventions, yet the consequences of this objective are not well understood. We develop a geometric perspective on intervention learning that characterizes intervention avoidance as constraining policies to a face of the occupancy measure polytope. This view reveals that the effectiveness of intervention learning depends on the informativeness of the intervention strategy: highly informative interventions uniquely determine the solution, while weak interventions leave a large set of feasible policies, many of which are suboptimal. Motivated by this under-specification, we define Robust Intervention Learning (RIL) as the problem of learning policies that perform well under varying levels of intervention informativeness. From the geometric formulation, we derive Residual Intervention Fine-Tuning (RIFT), which combines interventions with a prior policy to select among feasible solutions. We show that RIFT provides provable improvement over the prior and corresponds to solving a constrained reinforcement learning problem with an induced reward. Empirically, RIFT yields consistent policy improvement across a range of intervention settings, particularly when interventions are sparse or weakly informative. These results highlight the importance of accounting for intervention informativeness and suggest a principled path toward robust learning from human feedback.