Data to 7 Oct 2026

TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback

arXiv 2407.16574 ↗First posted 23 Jul 2024cs.CLcited by 43

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) leverages human preference data to train language models to align more closely with human essence. These human preference data, however, are labeled at the sequence level, creating a mismatch between sequence-level preference labels and tokens, which are autoregressively generated from the language model. Although several recent approaches have tried to provide token-level (i.e., dense) rewards for each individual token, these typically rely on predefined discrete reward values (e.g., positive: +1, negative: -1, neutral: 0), failing to account for varying degrees of preference inherent to each token. To address this limitation, we introduce TLCR (Token-Level Continuous Reward) for RLHF, which incorporates a discriminator trained to distinguish positive and negative tokens, and the confidence of the discriminator is used to assign continuous rewards to each token considering the context. Extensive experiments show that our proposed TLCR leads to consistent performance improvements over previous sequence-level or token-level discrete rewards on open-ended generation benchmarks.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 28 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

22 of 28 references.

Closest ancestors · 3

Same direction, newest first

Other references · 19

2022Training language models to follow instructions with human feedbackcited by 15,712
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2017Deep reinforcement learning from human preferencescited by 3,846
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 82

7 more

2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
2022Information-Theoretic Text Hallucination Reduction for Video-grounded Dialoguecited by 18
2023HEAR: Hearing Enhanced Audio Response for Video-grounded Dialoguecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 21; 2026: 14.

43 papers.

Sort:

By area:

2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 20
2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15
2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillationcited by 15
2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
2025Curiosity-Driven Reinforcement Learning from Human Feedbackcited by 14

31 more

2025SCAR: Shapley Credit Assignment for More Efficient RLHFcited by 13
2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
2024A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglementcited by 10
2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimizationcited by 10
2024SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Maskscited by 9
2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 8
2026Beyond Uniform Credit: Causal Credit Assignment for Policy Optimizationcited by 7
20242D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervisioncited by 6
2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preferencecited by 5
2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspectivecited by 5
2025Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Modelscited by 5
2025Harness Local Rewards for Global Benefits: Effective Text-to-Video Generation Alignment with Patch-level Reward Modelscited by 4
2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 4
2026PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoningcited by 3
2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
2025ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoningcited by 2
2025SGPO: Self-Generated Preference Optimization based on Self-Improvercited by 1
2026Explainable Token-level Noise Filtering for LLM Fine-tuning Datasetscited by 1
2026Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modelingcited by 1
2026BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Modelscited by 1
2025Learning Guarantee of Reward Modeling Using Deep Neural Networkscited by 0
2025Test-time reward-guided alignment of language models by importance sampling on pre-logit spacecited by 0
2025Towards better dense rewards in Reinforcement Learning Applicationscited by 0
2026rePIRL: Learn PRM with Inverse RL for LLM Reasoningcited by 0
2026ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Trainingcited by 0
2026Distribution Corrected Offline Data Distillation for Large Language Modelscited by 0
2026BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigationcited by 0
2026Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Updatecited by 0
2026S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHFcited by 0
2026Reward-Aware Population Scaling of Evolutionary Strategies in LLM Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way