mlsanityData to 14 Oct 2026

Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding

arXiv 2405.19763 ↗First posted 30 May 2024cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Recent strides in large language models (LLMs) have yielded remarkable performance, leveraging reinforcement learning from human feedback (RLHF) to significantly enhance generation and alignment capabilities. However, RLHF encounters numerous challenges, including the objective mismatch issue, leading to suboptimal performance in Natural Language Understanding (NLU) tasks. To address this limitation, we propose a novel Reinforcement Learning framework enhanced with Label-sensitive Reward (RLLR) to amplify the performance of LLMs in NLU tasks. By incorporating label-sensitive pairs into reinforcement learning, our method aims to adeptly capture nuanced label-sensitive semantic features during RL, thereby enhancing natural language understanding. Experiments conducted on five diverse foundation models across eight tasks showcase promising results. In comparison to Supervised Fine-tuning models (SFT), RLLR demonstrates an average performance improvement of 1.54%. Compared with RLHF models, the improvement averages at 0.69%. These results reveal the effectiveness of our method for LLMs in NLU tasks. Code and data available at: https://github.com/MagiaSN/ACL2024_RLLR.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 31 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

25 of 31 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback31 Oct
  2. 2023The History and Risks of Reinforcement Learning and Human Feedback20 Oct
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2016Generative Adversarial Imitation Learningcited by 1,597
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
6 more
  1. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  2. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  3. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  4. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 251
  5. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 108
  6. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 36

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 7
  2. 2025Learning Natural Language Constraints for Safe Reinforcement Learning of Language Agentscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  2. 2024Natural Language Reinforcement Learning
  3. 2026Reward Modeling from Natural Language Human Feedback
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  5. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  6. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models
  7. 2026Reinforcement Learning from Meta-Evaluation: Aligning Language Models Without Ground-Truth Labels
  8. 2024Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning