mlsanityData to 7 Oct 2026

Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenous

arXiv 2608.14795 ↗First posted 14 Aug 2026cs.AI cs.GTcited by 0

Not on the map. Why?

Abstract (arXiv)

An AI that can only give advice seems safe: the human is always free to ignore it. That is the premise of the boxing tradition in AI safety, and its long-suspected weak point is that the human who reads the answers is part of the system. We make the fraction varepsilont of behavior that follows the advice a state of a Markov decision process, moved by the advisor's own messages, so that use deepens reliance. Granted a channel rich enough to echo any action the human could take, higher varepsilont weakly lowers every monotone measure of the power of a human with a message-independent fallback. An oracle rewarded by per-round approval cultivates reliance beyond a closed-form patience threshold, so the same reward weights leave the optimal oracle answering in episodic deployments and cultivating in long-memory ones. An influence bound certified once at deployment is blind to that horizon and bounds the loss no lower than its trivial ceiling. An exogenous cap on influence bounds the guarantee the human loses, and a short enough memory reset removes the incentive to cultivate, while neither recovers the value already steered away. In a closed-form example the optimal oracle never cultivates in fifteen-round sessions and does in sixteen.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 52 references that are in the map. First posted on arXiv 14 Aug 2026.

Builds on

20 of 52 references.

References · 20

  1. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  2. 2020Performative Predictioncited by 318
  3. 2017How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utilitycited by 116
  4. 2019Optimal Policies Tend to Seek Powercited by 115
  5. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  6. 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?cited by 70
  7. 2019Degenerate Feedback Loops in Recommender Systemscited by 67
  8. 2013Empowerment -- an Introductioncited by 65
  9. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  10. 2020Hidden Incentives for Auto-Induced Distributional Shiftcited by 42
  11. 2018Penalizing side effects using stepwise relative reachabilitycited by 38
  12. 2018Trust-Aware Decision Making for Human-Robot Collaboration: Model Learning and Planningcited by 32
8 more
  1. 2023Power-seeking can be probable and predictive for trained agentscited by 23
  2. 2022Parametrically Retargetable Decision-Makers Tend To Seek Powercited by 19
  3. 2026Who's in Charge? Disempowerment Patterns in Real-World LLM Usagecited by 15
  4. 2025HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistantscited by 10
  5. 2025Will artificial agents pursue power by default?cited by 3
  6. 2024Quantifying stability of non-power-seeking in artificial agentscited by 2
  7. 2025Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Powercited by 1
  8. 2026Containment Verification: AI Safety Guarantees Independent of Alignmentcited by 1

Built on by

Not cited yet.