# CodeJudge: Evaluating Code Generation with Large Language Models

arXiv: https://arxiv.org/abs/2410.02184 · HTML: https://mlsanity.com/paper/2410.02184/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 20 of 20)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [SPoC: Search-based Pseudocode to Code](https://mlsanity.com/paper/1906.04908/) (2019)
- [Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow](https://mlsanity.com/paper/1805.08949/) (2018)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback](https://mlsanity.com/paper/2307.14936/) (2023)
- [Out of the BLEU: how should we assess quality of the Code Generation models?](https://mlsanity.com/paper/2208.03133/) (2022)
- [ICE-Score: Instructing Large Language Models to Evaluate Code](https://mlsanity.com/paper/2304.14317/) (2023)
- [Does BLEU Score Work for Code Migration?](https://mlsanity.com/paper/1906.04903/) (2019)

## Built on by (most cited first, 30 of 44)

- [ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code](https://mlsanity.com/paper/2506.02314/) (2025)
- [KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes](https://mlsanity.com/paper/2506.06541/) (2025)
- [CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks](https://mlsanity.com/paper/2507.10535/) (2025)
- [Code Comprehension then Auditing for Unsupervised LLM Evaluation](https://mlsanity.com/paper/2410.03131/) (2024)
- [MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation](https://mlsanity.com/paper/2502.12468/) (2025)
- [LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research](https://mlsanity.com/paper/2506.17335/) (2025)
- [Safety Aware Task Planning via Large Language Models in Robotics](https://mlsanity.com/paper/2503.15707/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code](https://mlsanity.com/paper/2503.00691/) (2025)
- [Rethinking the Evaluation of Secure Code Generation](https://mlsanity.com/paper/2503.15554/) (2025)
- [Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator](https://mlsanity.com/paper/2505.19236/) (2025)
- [DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models](https://mlsanity.com/paper/2601.11895/) (2026)
- [CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation](https://mlsanity.com/paper/2504.13472/) (2025)
- [Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators](https://mlsanity.com/paper/2505.18601/) (2025)
- [ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?](https://mlsanity.com/paper/2602.19594/) (2026)
- [Human-AI Experience in Integrated Development Environments: A Systematic Literature Review](https://mlsanity.com/paper/2503.06195/) (2025)
- [The Debugging Decay Index: Rethinking Debugging Strategies for Code LLMs](https://mlsanity.com/paper/2506.18403/) (2025)
- [Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specifications](https://mlsanity.com/paper/2508.12358/) (2025)
- [Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection](https://mlsanity.com/paper/2508.20766/) (2025)
- [Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair](https://mlsanity.com/paper/2510.03217/) (2025)
- [Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge](https://mlsanity.com/paper/2504.19730/) (2025)
- [Chain-of-Code Collapse: Reasoning Failures in LLMs via Adversarial Prompting in Code Generation](https://mlsanity.com/paper/2506.06971/) (2025)
- [Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement](https://mlsanity.com/paper/2603.00539/) (2026)
- [Agentic Code Reasoning](https://mlsanity.com/paper/2603.01896/) (2026)
- [Bridging LLM-Generated Code and Requirements: Reverse Generation technique and SBC Metric for Developer Insights](https://mlsanity.com/paper/2502.07835/) (2025)
- [Human-In-The-Loop Software Development Agents: Challenges and Future Directions](https://mlsanity.com/paper/2506.11009/) (2025)
- [CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation](https://mlsanity.com/paper/2505.19502/) (2025)
- [Multicalibration for LLM-based Code Generation](https://mlsanity.com/paper/2512.08810/) (2025)
- [Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model](https://mlsanity.com/paper/2602.07878/) (2026)
- [When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMs](https://mlsanity.com/paper/2606.28438/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown promising performance in code generation. However, how to reliably evaluate code generated by LLMs remains an unresolved problem. This paper presents CodeJudge, a code evaluation framework that leverages LLMs to evaluate the semantic correctness of generated code without the need for test cases. We investigate different ways to guide the LLM in performing "slow thinking" to arrive at an in-depth and reliable evaluation. We experimented with four LLMs as evaluators on four code generation datasets and five programming languages. The results show that CodeJudge significantly outperformed existing methods in most settings. Furthermore, compared with a SOTA GPT-3.5-based code evaluation method, CodeJudge achieved better results even when using a much smaller model, Llama-3-8B-Instruct. Our code and datasets are available on GitHub https://github.com/VichyTong/CodeJudge.