mlsanityData to 14 Oct 2026

Coffee: Boost Your Code LLMs by Fixing Bugs with Feedback

arXiv 2311.07215 ↗First posted 13 Nov 2023cs.CL cs.SEcited by 15

Not on the map. Why?

Abstract (arXiv)

Code editing is an essential step towards reliable program synthesis to automatically correct critical errors generated from code LLMs. Recent studies have demonstrated that closed-source LLMs (i.e., ChatGPT and GPT-4) are capable of generating corrective feedback to edit erroneous inputs. However, it remains challenging for open-source code LLMs to generate feedback for code editing, since these models tend to adhere to the superficial formats of feedback and provide feedback with misleading information. Hence, the focus of our work is to leverage open-source code LLMs to generate helpful feedback with correct guidance for code editing. To this end, we present Coffee, a collected dataset specifically designed for code fixing with feedback. Using this dataset, we construct CoffeePots, a framework for COde Fixing with FEEdback via Preference-Optimized Tuning and Selection. The proposed framework aims to automatically generate helpful feedback for code editing while minimizing the potential risk of superficial feedback. The combination of Coffee and CoffeePots marks a significant advancement, achieving state-of-the-art performance on HumanEvalFix benchmark. Codes and model checkpoints are publicly available at https://github.com/Lune-Blue/COFFEE.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

22 of 29 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
10 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 988
  2. 2021Measuring Coding Challenge Competence With APPScited by 902
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  4. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  6. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  7. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  8. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 111
  9. 2020Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesiscited by 39
  10. 2023The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Pythoncited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 4; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  2. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  3. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 60
  4. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  5. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 48
  6. 2024From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debuggingcited by 46
  7. 2024Direct Judgement Preference Optimizationcited by 33
  8. 2025Learning to Generate Unit Tests for Automated Debuggingcited by 17
  9. 2024RGD: Multi-LLM Based Agent Debugger via Refinement and Generation Guidancecited by 12
  10. 2024Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Codecited by 9
  11. 2024A Problem-Oriented Perspective and Anchor Verification for Code Optimizationcited by 4
  12. 2025Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationalecited by 3
3 more
  1. 2025SDLog: A Deep Learning Framework for Detecting Sensitive Information in Software Logscited by 2
  2. 2025Generating High-Quality Datasets for Code Editing via Open-Source Language Modelscited by 2
  3. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
  2. 2025Self-Bootstrapping Automated Program Repair: Using LLMs to Generate and Evaluate Synthetic Training Data for Bug Repair
  3. 2024An Empirical Study on the Code Refactoring Capability of Large Language Models
  4. 2026RepoZero: Can LLMs Generate a Code Repository from Scratch?
  5. 2025Large Language Model Critics for Execution-Free Evaluation of Code Changes
  6. 2026Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?
  7. 2025Improving LLM-Generated Code Quality with GRPO
  8. 2025ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback