# Codabench: Flexible, Easy-to-Use and Reproducible Benchmarking Platform

arXiv: https://arxiv.org/abs/2110.05802 · HTML: https://mlsanity.com/paper/2110.05802/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 26 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 6 of 6)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Representation Learning on Graphs: Methods and Applications](https://mlsanity.com/paper/1709.05584/) (2017)
- [Molecular geometry prediction using a deep generative graph neural network](https://mlsanity.com/paper/1904.00314/) (2019)
- [Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark](https://mlsanity.com/paper/1806.01427/) (2018)

## Built on by (most cited first, 30 of 66)

- [The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics](https://mlsanity.com/paper/2310.19792/) (2023)
- [Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"](https://mlsanity.com/paper/2409.16603/) (2024)
- [Overview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articles](https://mlsanity.com/paper/2408.08566/) (2024)
- [Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment](https://mlsanity.com/paper/2411.03417/) (2024)
- [AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance](https://mlsanity.com/paper/2506.03828/) (2025)
- [AIM 2025 Low-light RAW Video Denoising Challenge: Dataset, Methods and Results](https://mlsanity.com/paper/2508.16830/) (2025)
- [NTIRE 2026 The Second Challenge on Day and Night Raindrop Removal for Dual-Focused Images: Methods and Results](https://mlsanity.com/paper/2604.10634/) (2026)
- [Benchmarking tree species classification from proximally-sensed laser scanning data: introducing the FOR-species20K dataset](https://mlsanity.com/paper/2408.06507/) (2024)
- [NTIRE 2026 Rip Current Detection and Segmentation (RipDetSeg) Challenge Report](https://mlsanity.com/paper/2604.17070/) (2026)
- [AIM 2025 Rip Current Segmentation (RipSeg) Challenge Report](https://mlsanity.com/paper/2508.13401/) (2025)
- [Low Light Image Enhancement Challenge at NTIRE 2026](https://mlsanity.com/paper/2604.17669/) (2026)
- [Swin-LiteMedSAM: A Lightweight Box-Based Segment Anything Model for Large-Scale Medical Image Datasets](https://mlsanity.com/paper/2409.07172/) (2024)
- [AIM 2024 Sparse Neural Rendering Challenge: Dataset and Benchmark](https://mlsanity.com/paper/2409.15041/) (2024)
- [Robust Deepfake Detection, NTIRE 2026 Challenge: Report](https://mlsanity.com/paper/2604.24163/) (2026)
- [LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text](https://mlsanity.com/paper/2410.12064/) (2024)
- [Building Machine Learning Challenges for Anomaly Detection in Science](https://mlsanity.com/paper/2503.02112/) (2025)
- [1M-Deepfakes Detection Challenge](https://mlsanity.com/paper/2409.06991/) (2024)
- [Efficient MedSAMs: Segment Anything in Medical Images on Laptop](https://mlsanity.com/paper/2412.16085/) (2024)
- [ONERA's CRM WBPN database for machine learning activities, related regression challenge and first results](https://mlsanity.com/paper/2505.06265/) (2025)
- [CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset](https://mlsanity.com/paper/2507.22752/) (2025)
- [SLRTP2025 Sign Language Production Challenge: Methodology, Results, and Future Work](https://mlsanity.com/paper/2508.06951/) (2025)
- [LSVOS 2025 Challenge Report: Recent Advances in Complex Video Object Segmentation](https://mlsanity.com/paper/2510.11063/) (2025)
- [ICPR 2026 Competition on Low-Resolution License Plate Recognition](https://mlsanity.com/paper/2604.22506/) (2026)
- [DRL-STNet: Unsupervised Domain Adaptation for Cross-modality Medical Image Segmentation via Disentangled Representation Learning](https://mlsanity.com/paper/2409.18340/) (2024)
- [SemEval-2025 Task 1: AdMIRe -- Advancing Multimodal Idiomaticity Representation](https://mlsanity.com/paper/2503.15358/) (2025)
- [Overview of the CXR-LT 2026 Challenge: Multi-Center Long-Tailed and Zero Shot Chest X-ray Classification](https://mlsanity.com/paper/2602.22092/) (2026)
- [Overview of the PsyDefDetect Shared Task at BioNLP 2026: Detecting Levels of Psychological Defense Mechanisms in Supportive Conversations](https://mlsanity.com/paper/2605.24907/) (2026)
- [FAIR Universe HiggsML Uncertainty Dataset and Competition](https://mlsanity.com/paper/2410.02867/) (2024)
- [Open-World Panoptic Segmentation](https://mlsanity.com/paper/2412.12740/) (2024)
- [MVA 2025 Small Multi-Object Tracking for Spotting Birds Challenge: Dataset, Methods, and Results](https://mlsanity.com/paper/2507.12832/) (2025)

## Abstract (arXiv)

Obtaining standardized crowdsourced benchmark of computational methods is a major issue in data science communities. Dedicated frameworks enabling fair benchmarking in a unified environment are yet to be developed. Here we introduce Codabench, an open-source, community-driven platform for benchmarking algorithms or software agents versus datasets or tasks. A public instance of Codabench (https://www.codabench.org/) is open to everyone, free of charge, and allows benchmark organizers to compare fairly submissions, under the same setting (software, hardware, data, algorithms), with custom protocols and data formats. Codabench has unique features facilitating the organization of benchmarks flexibly, easily and reproducibly, such as the possibility of re-using templates of benchmarks, and supplying compute resources on-demand. Codabench has been used internally and externally on various applications, receiving more than 130 users and 2500 submissions. As illustrative use cases, we introduce 4 diverse benchmarks covering Graph Machine Learning, Cancer Heterogeneity, Clinical Diagnosis and Reinforcement Learning.