mlsanityData to 14 Oct 2026

What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

arXiv 2510.26202 ↗First posted 30 Oct 2025cs.CL cs.AIcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Human feedback can alter language models in unpredictable and undesirable ways, as practitioners lack a clear understanding of what feedback data encodes. While prior work studies preferences over certain attributes (e.g., length or sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce What's In My Human Feedback? (WIMHF), a method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7 datasets, WIMHF identifies a small number of human-interpretable features that account for the majority of the preference prediction signal achieved by black-box models. These features reveal a wide diversity in what humans prefer, and the role of dataset-level context: for example, users on Reddit prefer informality and jokes, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such as that LMArena users tend to vote against refusals, often in favor of toxic content. The learned features enable effective data curation: re-labeling the harmful examples in Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained personalization: on the Community Alignment dataset, we learn annotator-specific weights over subjective features that improve preference prediction. WIMHF provides a human-centered analysis method for practitioners to better understand and use preference data.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 66 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

45 of 66 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset13 Jul
  2. 2025HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation29 Apr
  3. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modeling20 Apr
  4. 2024Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets18 Nov
  5. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?18 Oct
  6. 2024Inverse Constitutional AI: Compressing Preferences into Principles2 Jun
  7. 2024Aligning to Thousands of Preferences via System Message Generalization28 May
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference7 Mar
  9. 2024A Roadmap to Pluralistic Alignment7 Feb
  10. 2023Compositional preference models for aligning LMs17 Oct
  11. 2023Human Feedback is not Gold Standard28 Sep
  12. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback9 Mar

Other references · 33

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  3. 2024Scaling and evaluating sparse autoencoderscited by 553
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  5. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  6. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  8. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  9. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 143
  10. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  11. 2024BatchTopK Sparse Autoencoderscited by 130
  12. 2023What's In My Big Data?cited by 129
21 more
  1. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  2. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  3. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  4. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 78
  5. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
  6. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 63
  7. 2024Axioms for AI Alignment from Human Feedbackcited by 52
  8. 2025Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactionscited by 52
  9. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
  10. 2025Large-Scale Data Selection for Instruction Tuningcited by 35
  11. 2024Aligning Language Models with Demonstrated Feedbackcited by 34
  12. 2022The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimizationcited by 29
  13. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 27
  14. 2025Sparse Autoencoders for Hypothesis Generationcited by 26
  15. 2025Copilot Arena: A Platform for Code LLM Evaluation in the Wildcited by 26
  16. 2024Disentangling Dense Embeddings with Sparse Autoencoderscited by 23
  17. 2020From Optimizing Engagement to Measuring Valuecited by 21
  18. 2025Position: Use Sparse Autoencoders to Discover Unknownscited by 19
  19. 2025PrefPalette: Personalized Preference Modeling with Latent Attributescited by 16
  20. 2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
  21. 2025BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 26.

26 papers.

Sort:

By area:

  1. 2026Shaping capabilities with token-level data filteringcited by 13
  2. 2026Automatically Finding Reward Model Biasescited by 4
  3. 2026Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Trainingcited by 4
  4. 2026What Makes Good Multilingual Reasoning? Disentangling Traces with Measurable Featurescited by 4
  5. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
  6. 2026The Measurement Revolution? Credible Measurement and Inference in the Age of AIcited by 3
  7. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  8. 2026Base Models Look Human To AI Detectorscited by 2
  9. 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modelingcited by 2
  10. 2026Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalcited by 2
  11. 2026Automated Concept Discovery for LLM-as-a-Judge Preference Analysiscited by 1
  12. 2026Multi-Perspective LLM Annotations for Valid Analyses in Subjective Taskscited by 1
14 more
  1. 2026Comparing Developer and LLM Biases in Code Evaluationcited by 1
  2. 2026RLHF May Not Reflect Genuine Preferencescited by 1
  3. 2026PRISM-X: Experiments on Personalised Fine-Tuning with Human and Simulated Userscited by 1
  4. 2026PLURAL: A Global Dataset for Value Alignmentcited by 1
  5. 2026Step-Level Preference Learning for Generative Agents in Social Simulationscited by 1
  6. 2026From Weights to Words: Expressing and Editing Preference Model Inferences in Natural Languagecited by 1
  7. 2026Disentangling Steering Vectorscited by 1
  8. 2026Understanding Annotator Safety Policy with Interpretabilitycited by 0
  9. 2026Coherence Maximization Improves Pluralistic Alignmentcited by 0
  10. 2026Open Problems in Constitutional Preference Reconstructioncited by 0
  11. 2026What do Reward Models Memorize?cited by 0
  12. 2026Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrastcited by 0
  13. 2026On the Role of Citations in Preference Datacited by 0
  14. 2026JudgeProfile: Understanding and Steering Subjectivity in LLM Judgescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values
  2. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
  3. 2024A Survey on Human Preference Learning for Large Language Models
  4. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback
  5. 2023DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4
  6. 2026Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
  7. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  8. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models