mlsanityData to 7 Oct 2026

Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders

arXiv 2605.16339 ↗First posted 7 May 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference learning in large language models relies on reward models as proxies for human judgment. However, these models frequently exhibit preference instability, producing contradictory preference assignments in response to subtle, meaning-preserving input variations. We analyze this instability at the representation level under three semantic-preserving perturbation types: paraphrasing, pattern injection, and backdoor triggers. We attribute this instability to over-reliance on predictive yet brittle features, which we term unstable features, and isolate them via Sparse Autoencoders (SAEs) in a sparse latent space where benign and perturbed inputs activate distinctly separable patterns. Building on this separability, we propose two SAE-based instability mitigation strategies: SAE Feature Steering, which identifies and suppresses anomalously activated features at inference, and SAE Residual Correction, which learns adaptive adjustments over SAE features to restore correct preferences. Our methods substantially reduce incorrect preference assignments on harmlessness and hallucination benchmarks while preserving benign performance and general utility on other tasks, without retraining the reward model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 44 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

38 of 44 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Interpretable Reward Model via Sparse Autoencoder12 Aug
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  3. 2025SAFER: Probing Safety in Reward Models with Sparse Autoencoder1 Jul
  4. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment16 Jan
  5. 2024Exploring and Addressing Reward Confusion in Offline Preference Learning22 Jul
  6. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  7. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  8. 2023The Trickle-down Impact of Reward (In-)consistency on RLHF28 Sep
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  10. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
2 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  8. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  10. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  11. 2022Toy Models of Superpositioncited by 928
  12. 2023Steering Language Models With Activation Engineeringcited by 824
12 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  3. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  4. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  5. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  6. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  7. 2022Polysemanticity and Capacity in Neural Networkscited by 74
  8. 2025Detecting Strategic Deception Using Linear Probescited by 65
  9. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignmentcited by 22
  10. 2024Preference Poisoning Attacks on Reward Model Learningcited by 14
  11. 2021Adversarial Visual Robustness by Causal Interventioncited by 13
  12. 2025Learning Safety Constraints for Large Language Modelscited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder
  2. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
  3. 2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
  4. 2026SteerRM: Debiasing Reward Models via Sparse Autoencoders
  5. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
  6. 2025Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modeling
  7. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  8. 2024Transforming and Combining Rewards for Aligning Large Language Models