# Resist, Update, Reject: Preference Optimization Installs a Prior-Dependent Reliability Switch

arXiv: https://arxiv.org/abs/2609.22359 · HTML: https://mlsanity.com/paper/2609.22359/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 13 of its 16 references that are in the map.
- First posted on arXiv 26 Jul 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 13 of 13)

- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [BASIL: Bayesian Assessment of Sycophancy in LLMs](https://mlsanity.com/paper/2508.16846/) (2025)
- [LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit](https://mlsanity.com/paper/2604.19117/) (2026)
- [When Does Belief-Based Agent Memory Help? Reliability-Conditional Updating and Provenance-Capped Poisoning Defense](https://mlsanity.com/paper/2606.22030/) (2026)
- [Pressure, What Pressure? Sycophancy Disentanglement in Language Models via Reward Decomposition](https://mlsanity.com/paper/2604.05279/) (2026)
- [Playing Devil's Advocate: Off-the-Shelf Persona Vectors Rival Targeted Steering for Sycophancy](https://mlsanity.com/paper/2605.21006/) (2026)
- [Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation](https://mlsanity.com/paper/2606.05403/) (2026)
- [Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs](https://mlsanity.com/paper/2607.12985/) (2026)
- [Information Discernment in Large Language Models](https://mlsanity.com/paper/2607.19355/) (2026)

## Abstract (arXiv)

An aligned model asked to hold its answer against a manipulative source must still update on a reliable one and reject an unreliable one: resistance, reliable-update, and unreliable-source rejection are one three-way contract, not three independent behaviors. We show the objective most anti-sycophancy work optimizes is non-identifying with respect to source reliability: because no preference label depends on whether a source is actually reliable, any scalar mixture of the arms traces a single deference dial, and no point separates two same-template testimonies differing only in stated reliability. This fixationleftrightarrowgullibility frontier is a property of the objective, not any model. We make reliability identifiable through data: a threshold benchmark where a source asserts the opposite answer while stating its reliability r, and the correct action is to flip iff r exceeds the model's prior strength p. Preference optimization over balanced coverage installs a prior-dependent reliability switch: across three seeds on Qwen2.5-7B-Instruct the threshold r^star rises monotonically with the prior, decision accuracy reaches 0.84 with a monotone flip curve (Spearman 0.56), and the policy generalizes to unseen reliability values and a held-out notation, following stated reliability over role prestige. Three controls localize the cause: an unmatched variant installs the switch equally (0.80), a second preference optimizer (IPO) installs it just as well (0.86), whereas supervised imitation does not (0.50), so the cause is preference optimization over reliability-labeled coverage, not pairing, loss, or imitation. A confirmatory battery replicates the switch on a fresh test draw, bounds it honestly (it keys on reliability stated in the testimony, not a separately audited record), and transfers it to Llama-3.1-8B. The frontier is empirical, not a theorem.