mlsanityData to 14 Oct 2026

ABC-Align: Prediction-Powered Alignment with Adaptive Bias Control

arXiv 2609.34374 ↗First posted 28 Sep 2026cs.LG cs.AI stat.MLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Language model post-training is often bottlenecked by the need for human-collected preference data, which is expensive and difficult to scale. Reinforcement learning from AI feedback (RLAIF) style approaches that leverage pseudo labels offer an abundant alternative but introduce systematic biases that degrade downstream alignment. Recent general-purpose semi-supervised methods correct for teacher bias using a small set of human-labeled examples, but suffer from high variance especially when human annotations are scarce. To this end, we propose ABC-Align, leveraging abundant pseudo label signal to minimize variance and applying a lightweight, adaptive correction grounded in the human-labeled subset. The correction strength is tuned automatically during training using plug-in estimates of the relevant bias--variance quantities. On LLM alignment with RLHF, DPO, and GRPO where human feedback is scarce, we empirically demonstrate that ABC-Align achieves superior performance over prior semi-supervised baselines in a series of experiments on an increasing scale. Our code is available at https://github.com/SewoongLab/abc-align .

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 91 of its 106 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

91 of 106 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Semi-Supervised Preference Optimization with Limited Feedback28 Oct
  2. 2025Doubly Robust Alignment for Large Language Models1 Jun
  3. 2025RLTHF: Targeted Human Feedback for LLM Alignment19 Feb
  4. 2024Your Weak LLM is Secretly a Strong Teacher for Alignment13 Sep
  5. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  6. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  7. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  8. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 81

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
38 more
  1. 2019Introduction to Online Convex Optimizationcited by 803
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  4. 2024Self-Rewarding Language Modelscited by 681
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  7. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  8. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 510
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  11. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  12. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  13. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  14. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  15. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  16. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  17. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  18. 2023Prediction-Powered Inferencecited by 220
  19. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  20. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  21. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  22. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  23. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  24. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  25. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
  26. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  27. 2023PPI++: Efficient Prediction-Powered Inferencecited by 117
  28. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  29. 2024How to Evaluate Reward Models for RLHFcited by 88
  30. 2020On the Convergence of SGD with Biased Gradientscited by 83
  31. 2022Debiased Self-Training for Semi-Supervised Learningcited by 62
  32. 2023Cross-Prediction-Powered Inferencecited by 58
  33. 2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 57
  34. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 51
  35. 2024Active Statistical Inferencecited by 50
  36. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 49
  37. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  38. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 41

50 most cited of 81 All 81 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  2. 2026Reliability-Aware LLM Alignment from Inconsistent Human Feedback
  3. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
  4. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
  5. 2025Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment
  6. 2025Aligning to What? Limits to RLHF Based Alignment
  7. 2023Aligning Large Language Models through Synthetic Feedback
  8. 2024Aligning Large Language Models via Fine-grained Supervision