# Benchopt: Reproducible, efficient and collaborative optimization benchmarks

arXiv: https://arxiv.org/abs/2206.13424 · HTML: https://mlsanity.com/paper/2206.13424/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 178 references that are in the map.
- First posted on arXiv 27 Jun 2022.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Minimizing Finite Sums with the Stochastic Average Gradient](https://mlsanity.com/paper/1309.2388/) (2013)
- [Benchmarking Simulation-Based Inference](https://mlsanity.com/paper/2101.04653/) (2021)
- [Revisiting ResNets: Improved Training and Scaling Strategies](https://mlsanity.com/paper/2103.07579/) (2021)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers](https://mlsanity.com/paper/2007.01547/) (2020)
- [Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence](https://mlsanity.com/paper/2002.04803/) (2020)
- [Regularizing Neural Networks via Adversarial Model Perturbation](https://mlsanity.com/paper/2010.04925/) (2020)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Adaptive piecewise polynomial estimation via trend filtering](https://mlsanity.com/paper/1304.2986/) (2013)
- [Gap Safe screening rules for sparsity enforcing penalties](https://mlsanity.com/paper/1611.05780/) (2016)

## Built on by (most cited first, 27 of 27)

- [Optimal Algorithms for Stochastic Bilevel Optimization under Relaxed Smoothness Conditions](https://mlsanity.com/paper/2306.12067/) (2023)
- [The largest EEG-based BCI reproducibility study for open science: the MOABB benchmark](https://mlsanity.com/paper/2404.15319/) (2024)
- [Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints](https://mlsanity.com/paper/2303.16510/) (2023)
- [A simple uniformly optimal method without line search for convex optimization](https://mlsanity.com/paper/2310.10082/) (2023)
- [A Lower Bound and a Near-Optimal Algorithm for Bilevel Empirical Risk Minimization](https://mlsanity.com/paper/2302.08766/) (2023)
- [94% on CIFAR-10 in 3.29 Seconds on a Single GPU](https://mlsanity.com/paper/2404.00498/) (2024)
- [What Do Machine Learning Researchers Mean by "Reproducible"?](https://mlsanity.com/paper/2412.03854/) (2024)
- [Experimenting with Normalization Layers in Federated Learning on non-IID scenarios](https://mlsanity.com/paper/2303.10630/) (2023)
- [Where Do Large Learning Rates Lead Us?](https://mlsanity.com/paper/2410.22113/) (2024)
- [Coordinate Descent for SLOPE](https://mlsanity.com/paper/2210.14780/) (2022)
- [A Meta-Learning Approach to Predicting Performance and Data Requirements](https://mlsanity.com/paper/2303.01598/) (2023)
- [Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods](https://mlsanity.com/paper/2508.12730/) (2025)
- [Convolutional Deep Kernel Machines](https://mlsanity.com/paper/2309.09814/) (2023)
- [Efficient Curvature-Aware Hypergradient Approximation for Bilevel Optimization](https://mlsanity.com/paper/2505.02101/) (2025)
- [A Second-Order Majorant Algorithm for Nonnegative Matrix Factorization](https://mlsanity.com/paper/2303.17992/) (2023)
- [Understanding Progressive Training Through the Framework of Randomized Coordinate Descent](https://mlsanity.com/paper/2306.03626/) (2023)
- [SKADA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation On Diverse Modalities](https://mlsanity.com/paper/2407.11676/) (2024)
- [Robust plug-and-play methods for highly accelerated non-Cartesian MRI reconstruction](https://mlsanity.com/paper/2411.01955/) (2024)
- [Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models](https://mlsanity.com/paper/2510.19999/) (2025)
- [Training neural networks faster with minimal tuning using pre-computed lists of hyperparameters for NAdamW](https://mlsanity.com/paper/2503.03986/) (2025)
- [Iterative Regularization with k-support Norm: An Important Complement to Sparse Recovery](https://mlsanity.com/paper/2401.05394/) (2023)
- [Fast Optimizer Benchmark](https://mlsanity.com/paper/2406.18701/) (2024)
- [Efficient Solvers for SLOPE in R, Python, Julia, and C++](https://mlsanity.com/paper/2511.02430/) (2025)
- [Second-Order, First-Class: A Composable Stack for Curvature-Aware Training](https://mlsanity.com/paper/2603.25976/) (2026)
- [Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Well](https://mlsanity.com/paper/2606.12552/) (2026)
- [An Efficient Newton Algorithm for Nonnegative Matrix Factorization with the Kullback-Leibler Divergence](https://mlsanity.com/paper/2607.13919/) (2026)
- [Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets](https://mlsanity.com/paper/2609.23995/) (2026)

## Abstract (arXiv)

Numerical validation is at the core of machine learning research as it allows to assess the actual impact of new methods, and to confirm the agreement between theory and practice. Yet, the rapid development of the field poses several challenges: researchers are confronted with a profusion of methods to compare, limited transparency and consensus on best practices, as well as tedious re-implementation work. As a result, validation is often very partial, which can lead to wrong conclusions that slow down the progress of research. We propose Benchopt, a collaborative framework to automate, reproduce and publish optimization benchmarks in machine learning across programming languages and hardware architectures. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments. To demonstrate its broad usability, we showcase benchmarks on three standard learning tasks: ℓ2-regularized logistic regression, Lasso, and ResNet18 training for image classification. These benchmarks highlight key practical findings that give a more nuanced view of the state-of-the-art for these problems, showing that for practical evaluation, the devil is in the details. We hope that Benchopt will foster collaborative work in the community hence improving the reproducibility of research findings.