Data to 7 Oct 2026

InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling

arXiv 2402.09345 ↗First posted 14 Feb 2024cs.LG cs.AIcited by 76

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Despite the success of reinforcement learning from human feedback (RLHF) in aligning language models with human values, reward hacking, also termed reward overoptimization, remains a critical challenge. This issue primarily arises from reward misgeneralization, where reward models (RMs) compute reward using spurious features that are irrelevant to human preferences. In this work, we tackle this problem from an information-theoretic perspective and propose a framework for reward modeling, namely InfoRM, by introducing a variational information bottleneck objective to filter out irrelevant information. Notably, we further identify a correlation between overoptimization and outliers in the IB latent space of InfoRM, establishing it as a promising tool for detecting reward overoptimization. Inspired by this finding, we propose the Cluster Separation Index (CSI), which quantifies deviations in the IB latent space, as an indicator of reward overoptimization to facilitate the development of online mitigation strategies. Extensive experiments on a wide range of settings and RM scales (70M, 440M, 1.4B, and 7B) demonstrate the effectiveness of InfoRM. Further analyses reveal that InfoRM's overoptimization detection mechanism is not only effective but also robust across a broad range of datasets, signifying a notable advancement in the field of RLHF. The code will be released upon acceptance.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 62 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

47 of 62 references.

Closest ancestors · 16

Same direction, newest first

4 more

Other references · 31

2022Training language models to follow instructions with human feedbackcited by 15,712
2017Proximal Policy Optimization Algorithmscited by 14,441
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
2020Learning to summarize from human feedbackcited by 2,562
2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
2016Deep Variational Information Bottleneckcited by 915
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906

19 more

2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
2015Deep Learning and the Information Bottleneck Principlecited by 793
2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
2023Baichuan 2: Open Large-scale Language Modelscited by 664
2019On Variational Bounds of Mutual Informationcited by 550
2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
2018Compressing Neural Networks using the Variational Information Bottleneckcited by 59
2023Won't Get Fooled Again: Answering Questions with False Premisescited by 47
2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
2020Understanding Learned Reward Functionscited by 37
2023BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformercited by 23
2023Improving Probability-based Prompt Selection Through Unified Evaluation and Analysiscited by 13
2022Information-Theoretic Odometry Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 40; 2026: 35.

76 papers.

Sort:

By area:

2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
2025Inference-Time Reward Hacking in Large Language Modelscited by 32
2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
2025Learning to Substitute Words with Model-based Score Rankingcited by 10
2025Reward Model Overoptimisation in Iterated RLHFcited by 10

38 more

2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Trainingcited by 9
2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 9
2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 8
2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Modelscited by 7
2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
2026How Many Tools Should an LLM Agent See? A Chance-Corrected Answercited by 7
2025SAFER: Probing Safety in Reward Models with Sparse Autoencodercited by 6
2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbackscited by 6
2026OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agentscited by 6
2025A Review of Incorporating Psychological Theories in LLMscited by 5
2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
2025Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?cited by 5
2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
2025AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenderscited by 5
2025Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillationcited by 5
2025Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correctioncited by 5
2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hackingcited by 5
2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modelingcited by 5
2026Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?cited by 5
2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 4
2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
2026IR3: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hackingcited by 4
2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 4
2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
2026Uncertainty-Aware Reward Modeling for Stable RLHFcited by 4
2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
2026Optimal Transport for LLM Reward Modeling from Noisy Preferencecited by 3
2026Discretizing Reward Modelscited by 3
2025BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHFcited by 2
2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
2026Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairingcited by 2
2026Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimizationcited by 2
2026ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignmentcited by 2

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way