# CodeArena: A Collective Evaluation Platform for LLM Code Generation

arXiv: https://arxiv.org/abs/2503.01295 · HTML: https://mlsanity.com/paper/2503.01295/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2025.

## Built on by (most cited first, 10 of 10)

- [ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces](https://mlsanity.com/paper/2604.05172/) (2026)
- [EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code](https://mlsanity.com/paper/2505.13004/) (2025)
- [Code Roulette: How Prompt Variability Affects LLM Code Generation](https://mlsanity.com/paper/2506.10204/) (2025)
- [ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenas](https://mlsanity.com/paper/2602.04296/) (2026)
- [PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization](https://mlsanity.com/paper/2605.15222/) (2026)
- [LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation](https://mlsanity.com/paper/2607.24780/) (2026)
- [DeepJSONEval: Benchmarking Complex Nested JSON Data Mining for Large Language Models](https://mlsanity.com/paper/2509.25922/) (2025)
- [Paper Espresso: From Paper Overload to Research Insight](https://mlsanity.com/paper/2604.04562/) (2026)
- [OpsLLM: Construction of Large Language Model for Software Operations with Multi-stage Learning](https://mlsanity.com/paper/2605.02906/) (2026)
- [Mastermind: Strategy-grounded Learning for Repository-Scale Vulnerability Reproduction](https://mlsanity.com/paper/2607.01764/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have reshaped code generation by synergizing their exceptional comprehension of natural language and programming syntax, thereby substantially boosting developer productivity. These advancements have prompted numerous efforts to quantitatively evaluate their coding capabilities. However, persistent challenges, such as benchmark leakage, data dissipation, and limited system accessibility, continue to impede a timely and accurate assessment. To address these limitations, we introduce CodeArena, an online evaluation framework tailored for LLM code generation. The key innovation is a collective evaluation mechanism, which dynamically recalibrates individual model scores based on the holistic performance of all participating models, mitigating score biases caused by widespread benchmark leakage. In addition, CodeArena ensures open access to all submitted solutions and test cases and provides automation-friendly APIs to streamline the code evaluation workflow. Our main contributions are: (1) a collective evaluation system for unbiased assessment, (2) a public repository of solutions and test cases, and (3) automation-ready APIs for seamless integration.