mlsanityData to 14 Oct 2026

RLTHF: Targeted Human Feedback for LLM Alignment

arXiv 2502.13417 ↗First posted 19 Feb 2025cs.CL cs.AI cs.LGcited by 8

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Fine-tuning large language models (LLMs) to align with user preferences is challenging due to the high cost of quality human annotations in Reinforcement Learning from Human Feedback (RLHF) and the generalizability limitations of AI Feedback. To address these challenges, we propose RLTHF, a human-AI hybrid framework that combines LLM-based initial alignment with selective human annotations to achieve full-human annotation alignment with minimal effort. RLTHF identifies hard-to-annotate samples mislabeled by LLMs using a reward model's reward distribution and iteratively enhances alignment by integrating strategic human corrections while leveraging LLM's correctly labeled samples. Evaluations on HH-RLHF and TL;DR datasets show that RLTHF reaches full-human annotation-level alignment with only 6-7% of the human annotation effort. Furthermore, models trained on RLTHF's curated datasets for downstream tasks outperform those trained on fully human-annotated datasets, underscoring the effectiveness of RLTHF.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 44 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

34 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
18 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  5. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  6. 2024Self-Rewarding Language Modelscited by 681
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  8. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  9. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  10. 2023Self-Alignment with Instruction Backtranslationcited by 151
  11. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  12. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  13. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  14. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
  15. 2024Self-Evolved Reward Learning for LLMscited by 16
  16. 2025R.I.P.: Better Models by Survival of the Fittest Promptscited by 13
  17. 2024Learning from Contrastive Prompts: Automated Optimization and Adaptationcited by 7
  18. 2024Autonomous Prompt Engineering in Large Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Reinforcement Learning from Human Feedbackcited by 58
  2. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  4. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  5. 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarkscited by 1
  6. 2026A New Strategy for Artificial Intelligence: Training Foundation Models Directly on Human Brain Datacited by 0
  7. 2026ABC-Align: Prediction-Powered Alignment with Adaptive Bias Controlcited by 0
  8. 2026SynAct: Synergistic Active Feedback Learning for LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
  2. 2023Preference Ranking Optimization for Human Alignment
  3. 2026Reliability-Aware LLM Alignment from Inconsistent Human Feedback
  4. 2024On The Global Convergence Of Online RLHF With Neural Parametrization
  5. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  6. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback
  8. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment