mlsanityData to 7 Oct 2026

CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?

arXiv 2408.10718 ↗First posted 20 Aug 2024cs.SE cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models (LLMs) have showcased impressive code generation capabilities, primarily evaluated through language-to-code benchmarks. However, these benchmarks may not fully capture a model's code understanding abilities. We introduce CodeJudge-Eval (CJ-Eval), a novel benchmark designed to assess LLMs' code understanding abilities from the perspective of code judging rather than code generation. CJ-Eval challenges models to determine the correctness of provided code solutions, encompassing various error types and compilation issues. By leveraging a diverse set of problems and a fine-grained judging system, CJ-Eval addresses the limitations of traditional benchmarks, including the potential memorization of solutions. Evaluation of 12 well-known LLMs on CJ-Eval reveals that even state-of-the-art models struggle, highlighting the benchmark's ability to probe deeper into models' code understanding abilities. Our codes and benchmark are available at urlhttps://github.com/CodeLLM-Research/CodeJudge-Eval.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

41 of 55 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  12. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
29 more
  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  3. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  4. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  5. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  6. 2021Measuring Coding Challenge Competence With APPScited by 879
  7. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  8. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  9. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  11. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  12. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294
  13. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  14. 2020Unsupervised Translation of Programming Languagescited by 262
  15. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  16. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  17. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  18. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  19. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  20. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  21. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 100
  22. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  23. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  24. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  25. 2024Large Language Models as Test Case Generators: Performance Evaluation and Enhancementcited by 27
  26. 2024The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?cited by 20
  27. 2018A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmarkcited by 17
  28. 2022Summarize and Generate to Back-translate: Unsupervised Translation of Programming Languagescited by 12
  29. 2024CommitBench: A Benchmark for Commit Message Generationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 17; 2026: 11.

29 papers.

Sort:

By area:

  1. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  2. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Researchcited by 29
  3. 2025Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Computecited by 25
  4. 2025Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluationcited by 18
  5. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  6. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  7. 2025TreeDiff: AST-Guided Code Generation with Diffusion LLMscited by 11
  8. 2025Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoningcited by 8
  9. 2025HoarePrompt: Structural Reasoning About Program Correctness in Natural Languagecited by 7
  10. 2025Efficient MAP Estimation of LLM Judgment Performance with Prior Transfercited by 4
  11. 2025Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judgecited by 4
  12. 2026Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineeringcited by 3
17 more
  1. 2026LLM Judges Can Be Too Generous When There Is No Reference Answercited by 3
  2. 2024Assessing the Impact of Conspiracy Theories Using Large Language Modelscited by 2
  3. 2025Refactoring with LLMs: Bridging Human Expertise and Machine Understandingcited by 2
  4. 2026Scaffold, Not Vocabulary? A Controlled, Two-Tier, Pre-Registered Study of a Popperian Code-Generation Skillcited by 2
  5. 2026When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMscited by 2
  6. 2025AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibrationcited by 1
  7. 2026Simple Role Assignment is Extraordinarily Effective for Safety Alignmentcited by 1
  8. 2026Agent4cs: A Multi-agent System for Code Summarization in Large Hierarchical Codebasescited by 1
  9. 2025ReCatcher: Towards LLMs Regression Testing for Code Generationcited by 0
  10. 2025NeuroDeX: Unlocking Diverse Support in Decompiling Deep Neural Network Executablescited by 0
  11. 2025GRASP: Graph Reasoning Agents for Systems Pharmacology with Human-in-the-Loopcited by 0
  12. 2025Beyond Blind Spots: Analytic Hints for Mitigating LLM-Based Evaluation Pitfallscited by 0
  13. 2026Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automationcited by 0
  14. 2026PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agentscited by 0
  15. 2026CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judgingcited by 0
  16. 2026AutoSG: LLM-Driven Solver Generation Solely from Task Prompts for Expensive Optimizationcited by 0
  17. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0