mlsanityData to 7 Oct 2026

CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences

arXiv 2403.09032 ↗First posted 14 Mar 2024cs.SE cs.CL cs.LGcited by 56

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Evaluating the alignment of large language models (LLMs) with user-defined coding preferences is a challenging endeavour that requires a deep assessment of LLMs' outputs. Existing methods and benchmarks rely primarily on automated metrics and static analysis tools, which often fail to capture the nuances of user instructions and LLM outputs. To address this gap, we propose using the LLM-as-a-Judge methodology to evaluate the alignment of LLMs with coding preferences. Based on this approach, we present CodeUltraFeedback, a comprehensive dataset designed to facilitate the evaluation and improvement of LLM alignment. CodeUltraFeedback consists of 10,000 coding instructions, each annotated with four responses generated from a diverse pool of 14 LLMs. These responses are ranked based on five distinct coding preferences using GPT-3.5 as a judge, providing both numerical scores and detailed textual feedback. Our analysis of CodeUltraFeedback reveals that responses from GPT-3.5 and GPT-4 are generally preferred over those from open-weight LLMs, highlighting significant differences in alignment between closed and open-weight models. In turn, we explore the usage of CodeUltraFeedback as feedback data to fine-tune and align CodeLlama-7B-Instruct using supervised fine-tuning (SFT) and reinforcement learning from AI feedback (RLAIF) with direct preference optimization (DPO). The resulting aligned CodeLlama-7B-Instruct model outperforms larger LLMs in terms of alignment with coding preferences and shows improved functional correctness on the HumanEval+ benchmark compared to the original instruct model. Therefore, our contributions bridge the gap in preference tuning of LLMs for code and set the stage for further advancements in model alignment and RLAIF in automated software engineering.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 75 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

53 of 75 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 49

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023Mistral 7Bcited by 3,065
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  12. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
37 more
  1. 2021Measuring Coding Challenge Competence With APPScited by 879
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  4. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  6. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  7. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  8. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  9. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  10. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  11. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 199
  12. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  13. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  14. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  15. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  16. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  17. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  18. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  19. 2023Shepherd: A Critic for Language Model Generationcited by 96
  20. 2023Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issuescited by 78
  21. 2023Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPTcited by 71
  22. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 62
  23. 2022XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligencecited by 54
  24. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  25. 2023RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repaircited by 50
  26. 2023Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4cited by 48
  27. 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
  28. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  29. 2024Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Codecited by 26
  30. 2023On the Evaluation of Neural Code Translation: Taxonomy and Benchmarkcited by 23
  31. 2024NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctnesscited by 23
  32. 2018A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmarkcited by 17
  33. 2023CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Modelscited by 16
  34. 2023FRANC: A Lightweight Framework for High-Quality Code Generationcited by 14
  35. 2023Generation-based Code Review Automation: How Far Are We?cited by 9
  36. 2024Improving the Learning of Code Review Successive Tasks with Cross-Task Knowledge Distillationcited by 7
  37. 2023Unity is Strength: Cross-Task Knowledge Distillation to Improve Code Review Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 25; 2026: 14.

56 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  4. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  5. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  6. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 64
  7. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  8. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
  9. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  10. 2026SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Taskscited by 45
  11. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  12. 2024Direct Judgement Preference Optimizationcited by 33
38 more
  1. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  2. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  3. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  4. 2024Learning Code Preference via Synthetic Evolutioncited by 19
  5. 2025MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Modelscited by 10
  6. 2024Aligning CodeLLMs with Direct Preference Optimizationcited by 9
  7. 2025Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQLcited by 9
  8. 2025Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluationcited by 9
  9. 2025AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMscited by 8
  10. 2025VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generationcited by 8
  11. 2024INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulnesscited by 7
  12. 2025R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillationcited by 7
  13. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoningcited by 7
  14. 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modelingcited by 7
  15. 2024SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Modelscited by 6
  16. 2025Combining Large Language Models with Static Analyzers for Code Review Generationcited by 6
  17. 2025OpenGenAlign: A Preference Dataset and Benchmark for Trustworthy Reward Modeling in Open-Ended, Long-Context Generationcited by 5
  18. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
  19. 2025Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specificationscited by 5
  20. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  21. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  22. 2025Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judgecited by 4
  23. 2026Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Modelcited by 4
  24. 2026Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgementcited by 4
  25. 2024Accelerating Direct Preference Optimization with Prefix Sharingcited by 3
  26. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  27. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  28. 2025Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluatorscited by 3
  29. 2025Retrieval-Augmented Fine-Tuning With Preference Optimization For Visual Program Generationcited by 2
  30. 2025Precise Attribute Intensity Control in Large Language Models via Targeted Representation Editingcited by 2
  31. 2025Leveraging Reward Models for Guiding Code Review Comment Generationcited by 1
  32. 2026Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuningcited by 1
  33. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  34. 2024WarriorCoder: Learning from Expert Battles to Augment Code Large Language Modelscited by 0
  35. 2025COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Valuescited by 0
  36. 2025Learning Bug Context for PyTorch-to-JAX Translation with LLMscited by 0
  37. 2025CodeAlignBench: Assessing Code Generation Models on Developer-Preferred Code Adjustmentscited by 0
  38. 2025Toward Explaining Large Language Models in Software Engineering Taskscited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  2. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  3. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
  4. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
  5. 2024CodecLM: Aligning Language Models with Tailored Synthetic Data
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  8. 2024ABC Align: Large Language Model Alignment for Safety & Accuracy