mlsanityData to 7 Oct 2026

CodeJudge: Evaluating Code Generation with Large Language Models

arXiv 2410.02184 ↗First posted 3 Oct 2024cs.LG cs.CL cs.SEcited by 41

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown promising performance in code generation. However, how to reliably evaluate code generated by LLMs remains an unresolved problem. This paper presents CodeJudge, a code evaluation framework that leverages LLMs to evaluate the semantic correctness of generated code without the need for test cases. We investigate different ways to guide the LLM in performing "slow thinking" to arrive at an in-depth and reliable evaluation. We experimented with four LLMs as evaluators on four code generation datasets and five programming languages. The results show that CodeJudge significantly outperformed existing methods in most settings. Furthermore, compared with a SOTA GPT-3.5-based code evaluation method, CodeJudge achieved better results even when using a much smaller model, Llama-3-8B-Instruct. Our code and datasets are available on GitHub https://github.com/VichyTong/CodeJudge.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 20 of its 37 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

20 of 37 references.

References · 20

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  6. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  9. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  11. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  12. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
8 more
  1. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  2. 2019SPoC: Search-based Pseudocode to Codecited by 176
  3. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  4. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  5. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  6. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  7. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  8. 2019Does BLEU Score Work for Code Migration?cited by 49

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 27; 2026: 13.

41 papers.

Sort:

By area:

  1. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 34
  2. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluationcited by 32
  3. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  4. 2024Code Comprehension then Auditing for Unsupervised LLM Evaluationcited by 31
  5. 2025KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakescited by 30
  6. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Researchcited by 29
  7. 2025Safety Aware Task Planning via Large Language Models in Roboticscited by 24
  8. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  9. 2025Rethinking the Evaluation of Secure Code Generationcited by 10
  10. 2025How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Codecited by 9
  11. 2025Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluatorcited by 9
  12. 2026DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Modelscited by 8
29 more
  1. 2025CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generationcited by 6
  2. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
  3. 2026ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?cited by 6
  4. 2025Human-AI Experience in Integrated Development Environments: A Systematic Literature Reviewcited by 5
  5. 2025The Debugging Decay Index: Rethinking Debugging Strategies for Code LLMscited by 5
  6. 2025Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specificationscited by 5
  7. 2025Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injectioncited by 5
  8. 2025Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repaircited by 5
  9. 2025Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judgecited by 4
  10. 2025Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generationcited by 4
  11. 2026Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgementcited by 4
  12. 2026Agentic Code Reasoningcited by 4
  13. 2025Bridging LLM-Generated Code and Requirements: Reverse Generation technique and SBC Metric for Developer Insightscited by 3
  14. 2025Human-In-The-Loop Software Development Agents: Challenges and Future Directionscited by 3
  15. 2025Multicalibration for LLM-based Code Generationcited by 2
  16. 2026Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Modelcited by 2
  17. 2026When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMscited by 2
  18. 2025CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluationcited by 1
  19. 2025Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPycited by 1
  20. 2026ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Modelscited by 1
  21. 2025FLOW-BENCH: Towards Conversational Generation of Enterprise Workflowscited by 0
  22. 2025Embedding Alignment in Code Generation for Audiocited by 0
  23. 2025Learning Bug Context for PyTorch-to-JAX Translation with LLMscited by 0
  24. 2026Mechanism-Faithful Queueing Simulation Model Translation with Large Language Model Supportcited by 0
  25. 2026CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judgingcited by 0
  26. 2026Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphscited by 0
  27. 2026Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation taskscited by 0
  28. 2026Interpreting and Steering for Safe and Correct Code Generationcited by 0
  29. 2026AG-CoT: Verified Algorithmic Traces for LLM Program Synthesis on Clifford Circuitscited by 0