mlsanityData to 7 Oct 2026

SteerRM: Debiasing Reward Models via Sparse Autoencoders

arXiv 2603.12795 ↗First posted 13 Mar 2026cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) are critical components of alignment pipelines, yet they exhibit biases toward superficial stylistic cues, preferring better-presented responses over semantically superior ones. Existing debiasing methods typically require retraining or architectural modifications, while direct activation suppression degrades performance due to representation entanglement. We propose SteerRM, the first training-free method for debiasing reward models using Sparse Autoencoder (SAE)-based interventions. SteerRM isolates stylistic effects using contrastive paired responses, identifies bias-related SAE features with a strength-stability criterion, and suppresses them at inference time. Across six reward models on RM-Bench, SteerRM improves Hard-split accuracy by 7.3 points on average while preserving overall performance. Results on a Gemma-based reward model and a controlled non-format bias further suggest generalization across RM architectures and bias types. We further find that format-related features are concentrated in shallow layers and transfer across models, revealing shared architecture-level bias encoding patterns. These results show that SAE-based interventions can mitigate reward-model biases without retraining, providing a practical and interpretable solution for alignment pipelines.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 44 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

32 of 44 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder11 Nov
  2. 2025Interpretable Reward Model via Sparse Autoencoder12 Aug
  3. 2025Reward Models in Deep Reinforcement Learning: A Survey18 Jun
  4. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  5. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  6. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  7. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  8. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  9. 2024Post-hoc Reward Calibration: A Case Study on Length Bias25 Sep
  10. 2024Quantile Regression for Distributional Reward Models in RLHF16 Sep
  11. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  12. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
5 more
  1. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  3. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  7. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  8. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  9. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  10. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  11. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 283
  12. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 118
3 more
  1. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 86
  2. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 58
  3. 2024Towards Unifying Interpretability and Control: Evaluation via Interventioncited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Patterning in Practice: Debiasing Reward Models with Susceptibilitiescited by 0
  2. 2026From Constitutions to Control: Interpretable Rewards for Aligning Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Debiasing Reward Models via Causally Motivated Inference-Time Intervention
  2. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  3. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking
  4. 2026Automatically Finding Reward Model Biases
  5. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
  6. 2026ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment
  7. 2025Debiasing Reward Models by Representation Learning with Guarantees
  8. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models