mlsanityData to 14 Oct 2026

Resist, Update, Reject: Preference Optimization Installs a Prior-Dependent Reliability Switch

arXiv 2609.22359 ↗First posted 26 Jul 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

An aligned model asked to hold its answer against a manipulative source must still update on a reliable one and reject an unreliable one: resistance, reliable-update, and unreliable-source rejection are one three-way contract, not three independent behaviors. We show the objective most anti-sycophancy work optimizes is non-identifying with respect to source reliability: because no preference label depends on whether a source is actually reliable, any scalar mixture of the arms traces a single deference dial, and no point separates two same-template testimonies differing only in stated reliability. This fixationleftrightarrowgullibility frontier is a property of the objective, not any model. We make reliability identifiable through data: a threshold benchmark where a source asserts the opposite answer while stating its reliability r, and the correct action is to flip iff r exceeds the model's prior strength p. Preference optimization over balanced coverage installs a prior-dependent reliability switch: across three seeds on Qwen2.5-7B-Instruct the threshold r^star rises monotonically with the prior, decision accuracy reaches 0.84 with a monotone flip curve (Spearman 0.56), and the policy generalizes to unseen reliability values and a held-out notation, following stated reliability over role prestige. Three controls localize the cause: an unmatched variant installs the switch equally (0.80), a second preference optimizer (IPO) installs it just as well (0.86), whereas supervised imitation does not (0.50), so the cause is preference optimization over reliability-labeled coverage, not pairing, loss, or imitation. A confirmatory battery replicates the switch on a fresh test draw, bounds it honestly (it keys on reliability stated in the testimony, not a separately audited record), and transfers it to Llama-3.1-8B. The frontier is empirical, not a theorem.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 16 references that are in the map. First posted on arXiv 26 Jul 2026.

Builds on

13 of 16 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs14 Jul
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 11

  1. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  4. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  5. 2025BASIL: Bayesian Assessment of Sycophancy in LLMscited by 13
  6. 2026LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuitcited by 10
  7. 2026When Does Belief-Based Agent Memory Help? Reliability-Conditional Updating and Provenance-Capped Poisoning Defensecited by 6
  8. 2026Pressure, What Pressure? Sycophancy Disentanglement in Language Models via Reward Decompositioncited by 4
  9. 2026Playing Devil's Advocate: Off-the-Shelf Persona Vectors Rival Targeted Steering for Sycophancycited by 4
  10. 2026Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluationcited by 1
  11. 2026Information Discernment in Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?
  2. 2026Available Guardrails: Certifying Selective Prediction across ML Systems
  3. 2026Learning When to Trust via Selective Context Preference Optimization
  4. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
  5. 2026Can Revealed Preferences Clarify LLM Alignment and Steering?
  6. 2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization
  7. 2026Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure
  8. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics