# Code Comprehension then Auditing for Unsupervised LLM Evaluation

arXiv: https://arxiv.org/abs/2410.03131 · HTML: https://mlsanity.com/paper/2410.03131/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [SPoC: Search-based Pseudocode to Code](https://mlsanity.com/paper/1906.04908/) (2019)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [DebugBench: Evaluating Debugging Capability of Large Language Models](https://mlsanity.com/paper/2401.04621/) (2024)
- [Does BLEU Score Work for Code Migration?](https://mlsanity.com/paper/1906.04903/) (2019)
- [CodeJudge: Evaluating Code Generation with Large Language Models](https://mlsanity.com/paper/2410.02184/) (2024)
- [Source Code Summarization in the Era of Large Language Models](https://mlsanity.com/paper/2407.07959/) (2024)

## Built on by (most cited first, 30 of 32)

- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [OpenCodeReasoning: Advancing Data Distillation for Competitive Coding](https://mlsanity.com/paper/2504.01943/) (2025)
- [lmgame-Bench: How Good are LLMs at Playing Games?](https://mlsanity.com/paper/2505.15146/) (2025)
- [Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models](https://mlsanity.com/paper/2502.19918/) (2025)
- [Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning](https://mlsanity.com/paper/2509.25300/) (2025)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?](https://mlsanity.com/paper/2506.08295/) (2025)
- [Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLM](https://mlsanity.com/paper/2505.13890/) (2025)
- [Wan-Image: Pushing the Boundaries of Generative Visual Intelligence](https://mlsanity.com/paper/2604.19858/) (2026)
- [ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning](https://mlsanity.com/paper/2512.13095/) (2025)
- [Improving Parallel Program Performance with LLM Optimizers via Agent-System Interfaces](https://mlsanity.com/paper/2410.15625/) (2024)
- [General Modular Harness for LLM Agents in Multi-Turn Gaming Environments](https://mlsanity.com/paper/2507.11633/) (2025)
- [Understanding and Steering the Cognitive Behaviors of Reasoning Models at Test-Time](https://mlsanity.com/paper/2512.24574/) (2025)
- [DNR Bench: Benchmarking Over-Reasoning in Reasoning LLMs](https://mlsanity.com/paper/2503.15793/) (2025)
- [KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation](https://mlsanity.com/paper/2505.14552/) (2025)
- [TableGPT-R1: Advancing Tabular Reasoning Through Reinforcement Learning](https://mlsanity.com/paper/2512.20312/) (2025)
- [SLR: Automated Synthesis for Scalable Logical Reasoning](https://mlsanity.com/paper/2506.15787/) (2025)
- [Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions](https://mlsanity.com/paper/2506.08234/) (2025)
- [Adversarial Training for Process Reward Models](https://mlsanity.com/paper/2511.22888/) (2025)
- [Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities](https://mlsanity.com/paper/2601.21937/) (2026)
- [Recursive Think-Answer Process for LLMs and VLMs](https://mlsanity.com/paper/2603.02099/) (2026)
- [SCI-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verification](https://mlsanity.com/paper/2606.04579/) (2026)
- [Control-R: Towards controllable test-time scaling](https://mlsanity.com/paper/2506.00189/) (2025)
- [OmniEduBench: A Comprehensive Chinese Benchmark for Evaluating Large Language Models in Education](https://mlsanity.com/paper/2510.26422/) (2025)
- [AdaTIR: Adaptive Tool-Integrated Reasoning via Difficulty-Aware Policy Optimization](https://mlsanity.com/paper/2601.14696/) (2026)
- [BAR Conjecture: the Feasibility of Inference Budget-Constrained LLM Services with Authenticity and Reasoning](https://mlsanity.com/paper/2507.23170/) (2025)
- [The Self-Execution Benchmark: Measuring LLMs' Attempts to Overcome Their Lack of Self-Execution](https://mlsanity.com/paper/2508.12277/) (2025)
- [From Long to Short: LLMs Excel at Trimming Own Reasoning Chains](https://mlsanity.com/paper/2509.06174/) (2025)
- [Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes](https://mlsanity.com/paper/2511.20680/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) for unsupervised code correctness evaluation have recently gained attention because they can judge if code runs as intended without requiring reference implementations or unit tests, which may be unavailable, sparse, or unreliable. However, most prior approaches condition LLM evaluators directly on the full code implementation, forcing the model to jointly infer program behavior and evaluate correctness in a single step. This entanglement leads to misinterpretations of code behavior and unreliable judgments. To mitigate this issue, we introduce CoCoA, an unsupervised Code Comprehension then Auditing framework that first comprehends functionality to generate a natural-language explanation. Then it evaluates task alignment based on this explanation. By sequentially sampling comprehension before evaluation, CoCoA improves the quality of inferred program behavior and enables the evaluator to focus on behavioral alignment rather than raw implementation details. Across multiple datasets, programming languages, and models, CoCoA achieves up to 68% increased F1 score and up to 20% increased accuracy over the best-performing baselines.