# ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code

arXiv: https://arxiv.org/abs/2409.10280 · HTML: https://mlsanity.com/paper/2409.10280/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2024.

## Built on by (most cited first, 19 of 19)

- [Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering](https://mlsanity.com/paper/2502.06193/) (2025)
- [Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation](https://mlsanity.com/paper/2502.17521/) (2025)
- [Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks](https://mlsanity.com/paper/2505.16901/) (2025)
- [SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development](https://mlsanity.com/paper/2505.16975/) (2025)
- [Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark](https://mlsanity.com/paper/2412.18573/) (2024)
- [RAMBO: Enhancing RAG-based Repository-Level Method Body Completion](https://mlsanity.com/paper/2409.15204/) (2024)
- [OSVBench: Benchmarking LLMs on Specification Generation Tasks for Operating System Verification](https://mlsanity.com/paper/2504.20964/) (2025)
- [Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents](https://mlsanity.com/paper/2602.22764/) (2026)
- [Assessing the Quality and Security of AI-Generated Code: A Quantitative Analysis](https://mlsanity.com/paper/2508.14727/) (2025)
- [Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware](https://mlsanity.com/paper/2505.05057/) (2025)
- [OSS-Bench: Benchmark Generator for Coding LLMs](https://mlsanity.com/paper/2505.12331/) (2025)
- [MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios](https://mlsanity.com/paper/2506.13824/) (2025)
- [Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation](https://mlsanity.com/paper/2506.22776/) (2025)
- [Optimizing LLM Code Suggestions: Feedback-Driven Timing with Lightweight State Bounds](https://mlsanity.com/paper/2511.18842/) (2025)
- [Evaluating Language Models on Cross-Language Code Functional Equivalence](https://mlsanity.com/paper/2608.23961/) (2026)
- [Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration](https://mlsanity.com/paper/2512.20159/) (2025)
- [AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation](https://mlsanity.com/paper/2601.04540/) (2026)
- [AutoBridge: Automating Smart Device Integration with Centralized Platform](https://mlsanity.com/paper/2507.23178/) (2025)
- [Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attacks](https://mlsanity.com/paper/2609.09865/) (2026)

## Abstract (arXiv)

In recent years, the application of large language models (LLMs) to code-related tasks has gained significant attention. However, existing evaluation benchmarks often focus on limited scenarios, such as code generation or completion, which do not reflect the diverse challenges developers face in real-world contexts. To address this, we introduce ComplexCodeEval, a benchmark designed to assess LCMs in various development tasks, including code generation, completion, API recommendation, and test case generation. It includes 3,897 Java samples and 7,184 Python samples from high-star GitHub repositories, each annotated with function signatures, docstrings, and API references to simulate real development environments. Our experiments across ten LCMs reveal that context improves performance and that data leakage can lead to overestimation, highlighting the need for more accurate evaluations.