# Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenous

arXiv: https://arxiv.org/abs/2608.14795 · HTML: https://mlsanity.com/paper/2608.14795/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 52 references that are in the map.
- First posted on arXiv 14 Aug 2026.

## Builds on (most cited first, 20 of 20)

- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [Feedback Loops With Language Models Drive In-Context Reward Hacking](https://mlsanity.com/paper/2402.06627/) (2024)
- [Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?](https://mlsanity.com/paper/2502.15657/) (2025)
- [Empowerment -- an Introduction](https://mlsanity.com/paper/1310.1863/) (2013)
- [Degenerate Feedback Loops in Recommender Systems](https://mlsanity.com/paper/1902.10730/) (2019)
- [On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback](https://mlsanity.com/paper/2411.02306/) (2024)
- [Hidden Incentives for Auto-Induced Distributional Shift](https://mlsanity.com/paper/2009.09153/) (2020)
- [Penalizing side effects using stepwise relative reachability](https://mlsanity.com/paper/1806.01186/) (2018)
- [Trust-Aware Decision Making for Human-Robot Collaboration: Model Learning and Planning](https://mlsanity.com/paper/1801.04099/) (2018)
- [Power-seeking can be probable and predictive for trained agents](https://mlsanity.com/paper/2304.06528/) (2023)
- [Parametrically Retargetable Decision-Makers Tend To Seek Power](https://mlsanity.com/paper/2206.13477/) (2022)
- [Who's in Charge? Disempowerment Patterns in Real-World LLM Usage](https://mlsanity.com/paper/2601.19062/) (2026)
- [HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants](https://mlsanity.com/paper/2509.08494/) (2025)
- [Will artificial agents pursue power by default?](https://mlsanity.com/paper/2506.06352/) (2025)
- [Quantifying stability of non-power-seeking in artificial agents](https://mlsanity.com/paper/2401.03529/) (2024)
- [Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Power](https://mlsanity.com/paper/2508.00159/) (2025)
- [Containment Verification: AI Safety Guarantees Independent of Alignment](https://mlsanity.com/paper/2605.09045/) (2026)

## Abstract (arXiv)

An AI that can only give advice seems safe: the human is always free to ignore it. That is the premise of the boxing tradition in AI safety, and its long-suspected weak point is that the human who reads the answers is part of the system. We make the fraction varepsilont of behavior that follows the advice a state of a Markov decision process, moved by the advisor's own messages, so that use deepens reliance. Granted a channel rich enough to echo any action the human could take, higher varepsilont weakly lowers every monotone measure of the power of a human with a message-independent fallback. An oracle rewarded by per-round approval cultivates reliance beyond a closed-form patience threshold, so the same reward weights leave the optimal oracle answering in episodic deployments and cultivating in long-memory ones. An influence bound certified once at deployment is blind to that horizon and bounds the loss no lower than its trivial ceiling. An exogenous cap on influence bounds the guarantee the human loses, and a short enough memory reset removes the incentive to cultivate, while neither recovers the value already steered away. In a closed-form example the optimal oracle never cultivates in fifteen-round sessions and does in sixteen.