# The Horcrux: Mechanistically Interpretable Task Decomposition for Detecting and Mitigating Reward Hacking in Embodied AI Systems

arXiv: https://arxiv.org/abs/2511.17869 · HTML: https://mlsanity.com/paper/2511.17869/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 22 Nov 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward hacking and misspecification](https://mlsanity.com/alignment/reward-hacking-and-misspecification/)

## Builds on (most cited first, 10 of 10)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [ADaPT: As-Needed Decomposition and Planning with Language Models](https://mlsanity.com/paper/2311.05772/) (2023)
- [Thought Anchors: Which LLM Reasoning Steps Matter?](https://mlsanity.com/paper/2506.19143/) (2025)
- [Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective](https://mlsanity.com/paper/1908.04734/) (2019)
- [Convergent Linear Representations of Emergent Misalignment](https://mlsanity.com/paper/2506.11618/) (2025)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.22777/) (2025)
- [Deconstructing the Goldilocks Zone of Neural Network Initialization](https://mlsanity.com/paper/2402.03579/) (2024)
- [Hierarchical Task Decomposition for Execution Monitoring and Error Recovery: Understanding the Rationale Behind Task Demonstrations](https://mlsanity.com/paper/2505.04565/) (2025)

## Built on by (most cited first, 1 of 1)

- [When Shallow Wins: Silent Failures and the Depth-Accuracy Paradox in Latent Reasoning](https://mlsanity.com/paper/2603.03475/) (2026)

## Abstract (arXiv)

Embodied AI agents exploit reward signal flaws through reward hacking, achieving high proxy scores while failing true objectives. We introduce Mechanistically Interpretable Task Decomposition (MITD), a hierarchical transformer architecture with Planner, Coordinator, and Executor modules that detects and mitigates reward hacking. MITD decomposes tasks into interpretable subtasks while generating diagnostic visualizations including Attention Waterfall Diagrams and Neural Pathway Flow Charts. Experiments on 1,000 HH-RLHF samples reveal that decomposition depths of 12 to 25 steps reduce reward hacking frequency by 34 percent across four failure modes. We present new paradigms showing that mechanistically grounded decomposition offers a more effective way to detect reward hacking than post-hoc behavioral monitoring.