mlsanityData to 14 Oct 2026

Interpretable Reward Modeling with Active Concept Bottlenecks

arXiv 2507.04695 ↗First posted 7 Jul 2025cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning through selective concept annotation. Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts. To make this framework efficient in low-supervision settings, we formalize an active learning strategy that dynamically acquires the most informative concept labels. We propose an acquisition function based on Expected Information Gain and show that it significantly accelerates concept learning without compromising preference accuracy. Evaluated on the UltraFeedback dataset, our method outperforms baselines in interpretability and sample efficiency, marking a step towards more transparent, auditable, and human-aligned reward models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 20 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

16 of 20 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Preference Learning for AI Alignment: a Causal Perspective6 Jun
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 11

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  4. 2020Concept Bottleneck Modelscited by 1,032
  5. 2022Interactive Concept Bottleneck Modelscited by 79
  6. 2023A Closer Look at the Intervention Procedure of Concept Bottleneck Modelscited by 64
  7. 2024Stochastic Concept Bottleneck Modelscited by 51
  8. 2024Efficient Exploration for LLMscited by 45
  9. 2024Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?cited by 28
  10. 2024Deep Bayesian Active Learning for Preference Modeling in Large Language Modelscited by 19
  11. 2025Measuring Leakage in Concept-Based Methods: An Information Theoretic Approachcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Post-hoc Stochastic Concept Bottleneck Modelscited by 0
  2. 2026CraftAlign: Feature-Grounded Evaluation and Revision Guidance for AI Storiescited by 0
  3. 2026From Constitutions to Control: Interpretable Rewards for Aligning Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling
  2. 2025Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling
  3. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoning
  4. 2026Scaling Reward Modeling without Human Supervision
  5. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
  6. 2025Reward Model Interpretability via Optimal and Pessimal Tokens
  7. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback
  8. 2026MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling