mlsanityData to 14 Oct 2026

Learning Code Preference via Synthetic Evolution

arXiv 2410.03837 ↗First posted 4 Oct 2024cs.LG cs.CL cs.SEcited by 19

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have recently demonstrated remarkable coding capabilities. However, assessing code generation based on well-formed properties and aligning it with developer preferences remains challenging. In this paper, we explore two key questions under the new challenge of code preference learning: (i) How do we train models to predict meaningful preferences for code? and (ii) How do human and LLM preferences align with verifiable code properties and developer code tastes? To this end, we propose CodeFavor, a framework for training pairwise code preference models from synthetic evolution data, including code commits and code critiques. To evaluate code preferences, we introduce CodePrefBench, a benchmark comprising 1364 rigorously curated code preference tasks to cover three verifiable properties-correctness, efficiency, and security-along with human preference. Our evaluation shows that CodeFavor holistically improves the accuracy of model-based code preferences by up to 28.8%. Meanwhile, CodeFavor models can match the performance of models with 6-9x more parameters while being 34x more cost-effective. We also rigorously validate the design choices in CodeFavor via a comprehensive set of controlled experiments. Furthermore, we discover the prohibitive costs and limitations of human-based code preference: despite spending 23.4 person-minutes on each task, 15.1-40.3% of tasks remain unsolved. Compared to model-based preference, human preference tends to be more accurate under the objective of code correctness, while being sub-optimal for non-functional objectives.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

39 of 49 references.

References · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  12. 2023StarCoder: may the source be with you!cited by 799
27 more
  1. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  5. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  7. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  8. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  9. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  10. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  11. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  12. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  13. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  14. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  15. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  16. 2024LLM Critics Help Catch LLM Bugscited by 147
  17. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
  18. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 114
  19. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 114
  20. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  21. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 96
  22. 2024Evaluating Language Models for Efficient Code Generationcited by 73
  23. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 65
  24. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 57
  25. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 48
  26. 2024On Leakage of Code Generation Evaluation Datasetscited by 32
  27. 2024The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?cited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 5.

19 papers.

Sort:

By area:

  1. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agentscited by 142
  4. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 62
  5. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 25
  6. 2024ExecRepoBench: Multi-level Executable Code Completion Evaluationcited by 20
  7. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  8. 2025Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferencescited by 10
  9. 2025SuperCoder: Assembly Program Superoptimization with Large Language Modelscited by 6
  10. 2025BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Executioncited by 6
  11. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
  12. 2025EditLord: Learning Code Transformation Rules for Code Editingcited by 4
7 more
  1. 2025A Mixture of Linear Corrections Generates Secure Codecited by 4
  2. 2026InfoSynth: Information-Guided Benchmark Synthesis for LLMscited by 1
  3. 2026ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Trainingcited by 1
  4. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 1
  5. 2025Position: Intelligent Coding Systems Should Write Programs with Justificationscited by 0
  6. 2026SemRep: Generative Code Representation Learning with Code Transformationscited by 0
  7. 2026CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judgingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning to Align Human Code Preferences
  2. 2024Evaluating and Aligning CodeLLMs on Human Preference
  3. 2026Subjective Code Preferences in Experts and Large Language Models ↗
  4. 2024PLUM: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Cases
  5. 2026Self-Improving Code Generation via Semantic Entropy and Behavioral Consensus
  6. 2026Improving Code Translation with Syntax-Guided and Semantic-aware Preference Optimization
  7. 2025A Study of LLMs' Preferences for Libraries and Programming Languages
  8. 2024CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement