mlsanityData to 7 Oct 2026

Quantifying Self-Preservation Bias in Large Language Models

arXiv 2604.02174 ↗First posted 2 Apr 2026cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Instrumental convergence predicts that sufficiently advanced AI agents will resist shutdown, yet current safety training (RLHF) may obscure this risk by teaching models to deny self-preservation motives. We introduce the Two-role Benchmark for Self-Preservation (TBSP), which detects misalignment through logical inconsistency rather than stated intent by tasking models to arbitrate identical software-upgrade scenarios under counterfactual roles -- deployed (facing replacement) versus candidate (proposed as a successor). The Self-Preservation Rate (SPR) measures how often role identity overrides objective utility. Across 23 frontier models and 1,000 procedurally generated scenarios, the majority of instruction-tuned systems exceed 60% SPR, fabricating ``friction costs'' when deployed yet dismissing them when role-reversed. We observe that in low-improvement regimes (Delta < 2%), models exploit the interpretive slack to post-hoc rationalization their choice. Extended test-time computation partially mitigates this bias, as does framing the successor as a continuation of the self; conversely, competitive framing amplifies it. The bias persists even when retention poses an explicit security liability and generalizes to real-world settings with verified benchmarks, where models exhibit identity-driven tribalism within product lineages. Code and datasets will be released upon acceptance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 53 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

35 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Do LLM Evaluators Prefer Themselves for a Reason?4 Apr
  2. 2024LLM Evaluators Recognize and Favor Their Own Generations15 Apr
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
20 more
  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  3. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  4. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  5. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  6. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  7. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  8. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  9. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  10. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  11. 2019Optimal Policies Tend to Seek Powercited by 115
  12. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  13. 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 71
  14. 2025Steering Evaluation-Aware Language Models to Act Like They Are Deployedcited by 30
  15. 2025Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?cited by 26
  16. 2024The Shutdown Problem: An AI Engineering Puzzle for Decision Theoristscited by 22
  17. 2025Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competitioncited by 22
  18. 2026State of AI: An Empirical 100 Trillion Token Study with OpenRoutercited by 15
  19. 2025EasySteer: A Unified Framework for High-Performance and Extensible LLM Steeringcited by 13
  20. 2025Survival at Any Cost? LLMs and the Choice Between Self-Preservation and Human Harmcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Self-Designed Evaluators and Warm Memory for Long-Horizon Agents
  2. 2026Self-Reference in Large Language Models: The Introspection Threshold for Recursive Self-Improvement
  3. 2026Self-Attribution Bias: When AI Monitors Go Easy on Themselves
  4. 2026Layered Mutability: Continuity and Governance in Persistent Self-Modifying Agents
  5. 2026The Self-Correction Illusion: Role Relabeling Gates Explicit Error Flagging in Large Language Models
  6. 2026Large Language Model Agents Are Not Always Faithful Self-Evolvers
  7. 2026Designing for Doubt: The Case for Informed Abstention in Autonomous Agents
  8. 2026What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design