mlsanityData to 7 Oct 2026

CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models

arXiv 2403.02745 ↗First posted 5 Mar 2024cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

This paper addresses the challenges of aligning large language models (LLMs) with human values via preference learning (PL), focusing on incomplete and corrupted data in preference datasets. We propose a novel method for robustly and completely recalibrating values within these datasets to enhance LLMs' resilience against the issues. In particular, we devise a guaranteed polynomial time ranking algorithm that robustifies several existing models, such as the classic Bradley-Terry-Luce (BTL) (Bradley and Terry, 1952) model and certain generalizations of it. To the best of our knowledge, our present work is the first to propose an algorithm that provably recovers an ε-optimal ranking with high probability while allowing as large as O(n) perturbed pairwise comparison results per model response. Furthermore, we show robust recovery results in the partially observed setting. Our experiments confirm that our algorithms handle adversarial noise and unobserved comparisons well in both general and LLM preference dataset settings. This work contributes to the development and scaling of more reliable and ethically aligned AI models by equipping the dataset curation pipeline with the ability to handle missing and maliciously manipulated inputs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 117 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

53 of 117 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  2. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  3. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  4. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  7. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
33 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2024Mixtral of Expertscited by 1,785
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2024Self-Rewarding Language Modelscited by 654
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  9. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  10. 2023The Falcon Series of Open Language Modelscited by 526
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  12. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  13. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  14. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  15. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  16. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  17. 2023Large Language Model Guided Tree-of-Thoughtcited by 218
  18. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
  19. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  20. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  21. 2022Estimating the Personality of White-Box Language Modelscited by 47
  22. 2014Non-convex Robust PCAcited by 41
  23. 2016Fast Algorithms for Robust PCA via Gradient Descentcited by 39
  24. 2017Robust Subspace Learning: Robust PCA, Robust Subspace Tracking, and Robust Subspace Recoverycited by 25
  25. 2015Sync-Rank: Robust Ranking, Constrained Ranking and Rank Aggregation via Eigenvector and Semidefinite Programming Synchronizationcited by 14
  26. 2022A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learningcited by 12
  27. 2022On the identifiability of mixtures of ranking modelscited by 7
  28. 2023h2oGPT: Democratizing Large Language Modelscited by 5
  29. 2017Inductive Pairwise Ranking: Going Beyond the n log(n) Barriercited by 3
  30. 2022Self-Aware Feedback-Based Self-Learning in Large-Scale Conversational AIcited by 3
  31. 2022PENTATRON: PErsonalized coNText-Aware Transformer for Retrieval-based cOnversational uNderstandingcited by 3
  32. 2017Provable Inductive Robust PCA via Iterative Hard Thresholdingcited by 1
  33. 2023User Friendly and Adaptable Discriminative AI: Using the Lessons from the Success of LLMs and Image Generation Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2025HPS: Hard Preference Sampling for Human Preference Alignment
  4. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  5. 2025Robust Preference Alignment via Directional Neighborhood Consensus
  6. 2023Preference Ranking Optimization for Human Alignment
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2025ComPO: Preference Alignment via Comparison Oracles