mlsanityData to 7 Oct 2026

The Horcrux: Mechanistically Interpretable Task Decomposition for Detecting and Mitigating Reward Hacking in Embodied AI Systems

arXiv 2511.17869 ↗First posted 22 Nov 2025cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Embodied AI agents exploit reward signal flaws through reward hacking, achieving high proxy scores while failing true objectives. We introduce Mechanistically Interpretable Task Decomposition (MITD), a hierarchical transformer architecture with Planner, Coordinator, and Executor modules that detects and mitigates reward hacking. MITD decomposes tasks into interpretable subtasks while generating diagnostic visualizations including Attention Waterfall Diagrams and Neural Pathway Flow Charts. Experiments on 1,000 HH-RLHF samples reveal that decomposition depths of 12 to 25 steps reduce reward hacking frequency by 34 percent across four failure modes. We present new paradigms showing that mechanistically grounded decomposition offers a more effective way to detect reward hacking than post-hoc behavioral monitoring.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 10 of its 18 references that are in the map. First posted on arXiv 22 Nov 2025.

Builds on

10 of 18 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning28 Jun
  2. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation14 Mar

Other references · 8

  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  3. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
  4. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  5. 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
  6. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  7. 2024Deconstructing the Goldilocks Zone of Neural Network Initializationcited by 4
  8. 2025Hierarchical Task Decomposition for Execution Monitoring and Error Recovery: Understanding the Rationale Behind Task Demonstrationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026When Shallow Wins: Silent Failures and the Depth-Accuracy Paradox in Latent Reasoningcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale
  2. 2026From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents
  3. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking
  4. 2026Monitoring Emergent Reward Hacking During Generation via Internal Activations
  5. 2026Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
  6. 2026Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalization
  7. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
  8. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR