mlsanityData to 7 Oct 2026

Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision

arXiv 2501.07886 ↗First posted 14 Jan 2025cs.LG cs.AI cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Language model (LM) post-training relies on two stages of human supervision: task demonstrations for supervised finetuning (SFT), followed by preference comparisons for reinforcement learning from human feedback (RLHF). As LMs become more capable, the tasks they are given become harder to supervise. Will post-training remain effective under unreliable supervision? To test this, we simulate unreliable demonstrations and comparison feedback using small LMs and time-constrained humans. We find that in the presence of unreliable supervision, SFT still retains some effectiveness, but DPO (a common RLHF algorithm) fails to improve the model beyond SFT. To address this, we propose iterative label refinement (ILR) as an alternative to RLHF. ILR improves the SFT data by using comparison feedback to decide whether human demonstrations should be replaced by model-generated alternatives, then retrains the model via SFT on the updated data. SFT+ILR outperforms SFT+DPO on several tasks with unreliable supervision (math, coding, and safe instruction-following). Our findings suggest that as LMs are used for complex tasks where human supervision is unreliable, RLHF may no longer be the best use of human comparison feedback; instead, it is better to direct feedback towards improving the training data rather than continually training the model. Our code and data are available at https://github.com/helloelwin/iterative-label-refinement.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 58 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

46 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2018Scalable agent alignment via reward modeling: a research direction19 Nov

Other references · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2023Mistral 7Bcited by 3,065
  12. 2016Concrete Problems in AI Safetycited by 1,582
30 more
  1. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  4. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2023Instruction Tuning with GPT-4cited by 676
  7. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  8. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  9. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  10. 2018AI safety via debatecited by 384
  11. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  12. 2022Self-critiquing models for assisting human evaluatorscited by 372
  13. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  14. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  15. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  16. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  17. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  18. 2017Deep Learning is Robust to Massive Label Noisecited by 164
  19. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  20. 2018Supervising strong learners by amplifying weak expertscited by 139
  21. 2024LLM Critics Help Catch LLM Bugscited by 138
  22. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  23. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  24. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  25. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  26. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  27. 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 63
  28. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  29. 2023Debate Helps Supervise Unreliable Expertscited by 39
  30. 2024Bias Amplification in Language Model Evolution: An Iterated Learning Perspectivecited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  2. 2026Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversightcited by 5
  3. 2025Weak-to-Strong Generalization via Bregman Bias-Variance Decompositioncited by 3
  4. 2026GSDrive: Reinforcing Driving Policies by Multi-mode Future Trajectory Probing with 3D Gaussian Splatting Environmentcited by 0
  5. 2026On the Blessing of Pre-training in Weak-to-Strong Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Training Language Models with Language Feedback at Scale
  2. 2023SuperHF: Supervised Iterative Learning from Human Feedback
  3. 2024Aligning Language Models with Demonstrated Feedback
  4. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers
  5. 2024Reward Modeling with Weak Supervision for Language Models
  6. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Models
  7. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuning
  8. 2025REFINE-AF: A Task-Agnostic Framework to Align Language Models via Self-Generated Instructions using Reinforcement Learning from Automated Feedback