Data to 14 Oct 2026

Dense Reward for Free in Reinforcement Learning from Human Feedback

arXiv 2402.00782 ↗First posted 1 Feb 2024cs.LGcited by 78

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more

Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has been credited as the key advance that has allowed Large Language Models (LLMs) to effectively follow instructions and produce useful assistance. Classically, this involves generating completions from the LLM in response to a query before using a separate reward model to assign a score to the full completion. As an auto-regressive process, the LLM has to take many "actions" (selecting individual tokens) and only receives a single, sparse reward at the end of an episode, a setup that is known to be difficult to optimise in traditional reinforcement learning. In this work we leverage the fact that the reward model contains more information than just its scalar output, in particular, it calculates an attention map over tokens as part of the transformer architecture. We use these attention weights to redistribute the reward along the whole completion, effectively densifying the signal and highlighting the most important tokens, all without incurring extra computational cost or requiring any additional modelling. We demonstrate that, theoretically, this approach is equivalent to potential-based reward shaping, ensuring that the optimal policy remains unchanged. Empirically, we show that it stabilises training, accelerates the rate of learning, and, in practical cases, may lead to better local optima.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 79 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

48 of 79 references.

Closest ancestors · 10

Same direction, newest first

Other references · 38

2017Attention Is All You Needcited by 60,361
2020Language Models are Few-Shot Learnerscited by 32,419
2017Proximal Policy Optimization Algorithmscited by 15,215
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2023Let's Verify Step by Stepcited by 3,789
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
2013Playing Atari with Deep Reinforcement Learningcited by 3,274
2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
2020Learning to summarize from human feedbackcited by 2,588

26 more

2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
2017Learning Important Features Through Propagating Activation Differencescited by 1,458
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
2022Solving math word problems with process- and outcome-based feedbackcited by 901
2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
2021Recursively Summarizing Books with Human Feedbackcited by 290
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
2020Toward the Fundamental Limits of Imitation Learningcited by 96
2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
2019Hindsight Credit Assignmentcited by 79
2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 72
2020Reinforcement Learning with Random Delayscited by 59
2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 59
2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
2020Learning Guidance Rewards with Trajectory-space Smoothingcited by 30
2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 27
2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
2022POETREE: Interpretable Policy Learning with Adaptive Decision Treescited by 14
2020Bandit Learning with Delayed Impact of Actionscited by 10
2023Neural Laplace Control for Continuous-time Delayed Systemscited by 10
2022Inverse Online Learning: Understanding Non-Stationary and Reactionary Policiescited by 6
2023Optimising Human-AI Collaboration by Learning Convincing Explanationscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 34; 2026: 23.

78 papers.

Sort:

By area:

2026Reinforcement Learning via Self-Distillationcited by 332
2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 42
2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Modelscited by 27

38 more

2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignmentcited by 26
2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
2024A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directionscited by 25
2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 19
2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillationcited by 17
2025SCAR: Shapley Credit Assignment for More Efficient RLHFcited by 16
2025Discriminative Policy Optimization for Token-Level Reward Modelscited by 16
2025Curiosity-Driven Reinforcement Learning from Human Feedbackcited by 15
2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
2025TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewardscited by 14
2024Supervised Fine-Tuning as Inverse Reinforcement Learningcited by 12
2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
2025Doubly Robust Alignment for Large Language Modelscited by 12
2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
2024RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistributioncited by 10
2025MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modelingcited by 10
2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMscited by 10
2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 10
2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedbackcited by 9
2024Interpreting Language Reward Models via Contrastive Explanationscited by 9
2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimizationcited by 7
2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
2026Process-Verified Reinforcement Learning for Theorem Proving via Leancited by 7
2025Align to Structure: Aligning Large Language Models with Structural Informationcited by 6
2026Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoningcited by 6
2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
2024CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learningcited by 5
2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 5
2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preferencecited by 5
2025Attention-Based Reward Shaping for Sparse and Delayed Rewardscited by 5
2025OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Modelscited by 5
2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspectivecited by 5
2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoningcited by 5
2026Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energycited by 5
2024Large Language Model-Enhanced Reinforcement Learning for Generic Bus Holding Control Strategiescited by 4
2025Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shapingcited by 3
2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way