mlsanityData to 7 Oct 2026

Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code

arXiv 2409.19715 ↗First posted 29 Sep 2024cs.CLcited by 9

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

This paper presents Coffee-Gym, a comprehensive RL environment for training models that provide feedback on code editing. Coffee-Gym includes two major components: (1) Coffee, a dataset containing humans' code edit traces for coding questions and machine-written feedback for editing erroneous code; (2) CoffeeEval, a reward function that faithfully reflects the helpfulness of feedback by assessing the performance of the revised code in unit tests. With them, Coffee-Gym addresses the unavailability of high-quality datasets for training feedback models with RL, and provides more accurate rewards than the SOTA reward model (i.e., GPT-4). By applying Coffee-Gym, we elicit feedback models that outperform baselines in enhancing open-source code LLMs' code editing, making them comparable with closed-source LLMs. We make the dataset and the model checkpoint publicly available.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 43 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

25 of 43 references.

References · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2023StarCoder: may the source be with you!cited by 790
13 more
  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  4. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  5. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  6. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  7. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  8. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  9. 2024DebugBench: Evaluating Debugging Capability of Large Language Modelscited by 74
  10. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  11. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 60
  12. 2024CodeEditorBench: Evaluating Code Editing Capability of Large Language Modelscited by 22
  13. 2023Coffee: Boost Your Code LLMs by Fixing Bugs with Feedbackcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 4.

9 papers.

  1. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 116
  2. 2024Evaluating Robustness of Reward Models for Mathematical Reasoningcited by 21
  3. 2025Learning to Generate Unit Tests for Automated Debuggingcited by 16
  4. 2025ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programmingcited by 16
  5. 2025CADReview: Automatically Reviewing CAD Programs with Error Detection and Correctioncited by 3
  6. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  7. 2026Towards Direct Evaluation of Harness Optimizers via Priority Rankingcited by 2
  8. 2026Learning from 53.6K Real-World Developer Edits of AI-Generated Codecited by 1
  9. 2026Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarkingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback
  2. 2026Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming
  3. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
  4. 2023Improving Code Generation by Training with Natural Language Feedback
  5. 2025CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models
  6. 2025FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
  7. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  8. 2026SAFEdit: Does Multi-Agent Decomposition Resolve the Reliability Challenges of Instructed Code Editing?