mlsanityData to 7 Oct 2026

Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluation

arXiv 2412.07429 ↗First posted 10 Dec 2024cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Automatic evaluation by large language models (LLMs) is a prominent topic today; however, judgment and evaluation tasks are often subjective and influenced by various factors, making adaptation challenging. While many studies demonstrate the capabilities of state-of-the-art proprietary LLMs in comparison to human evaluators, they often struggle to adapt to reference evaluators over time, a requirement for achieving personalized judgment. Additionally, numerous works have attempted to apply open LLMs as judges or evaluators, but these efforts frequently overlook the limitations of working with scarce data. Personalized judgment is inherently associated with limited data scenarios, which are common in many real-world problems. Our work aims to present a data augmentation technique to select a more effective sample from limited data in order to align an open LLM with human preference. Our work achieves approximately 7% improvements in Pearson correlation with a reference judge over the baseline,and 30% improvement over the base model (Llama3.1-8B-Instruct) in the mathematical reasoning evaluation task. demonstrating that augmenting selecting more effective preference data enables our approach to surpass baseline methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 33 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

25 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Direct Judgement Preference Optimization23 Sep
  2. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback2 Oct

Other references · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2024Mixtral of Expertscited by 1,785
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  11. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  12. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
11 more
  1. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  2. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  3. 2023Challenges and Applications of Large Language Modelscited by 251
  4. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  5. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  6. 2023Fine-tuning ChatGPT for Automatic Scoringcited by 50
  7. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  8. 2024Data curation via joint example selection further accelerates multimodal learningcited by 32
  9. 2023Aligning Large Language Models with Human: A Surveycited by 31
  10. 2023Using GPT-4 to Augment Unbalanced Data for Automatic Scoringcited by 20
  11. 2023Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessmentscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Black-box Language Models with Human Judgments
  2. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
  3. 2024Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences
  4. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
  5. 2026DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration
  6. 2025ALIGN: Prompt-based Attribute Alignment for Reliable, Responsible, and Personalized LLM-based Decision-Making
  7. 2024Dissecting Human and LLM Preferences
  8. 2024Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessments