# Common Task Framework For a Critical Evaluation of Scientific Machine Learning Algorithms

arXiv: https://arxiv.org/abs/2510.23166 · HTML: https://mlsanity.com/paper/2510.23166/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2025.

## Built on by (most cited first, 13 of 13)

- [AI Must Embrace Specialization via Superhuman Adaptable Intelligence](https://mlsanity.com/paper/2602.23643/) (2026)
- [When is a System Discoverable from Data? Discovery Requires Chaos](https://mlsanity.com/paper/2511.08860/) (2025)
- [The Seismic Wavefield Common Task Framework](https://mlsanity.com/paper/2512.19927/) (2025)
- [CTF4Nuclear: Common Task Framework for Nuclear Fission and Fusion Models](https://mlsanity.com/paper/2605.15549/) (2026)
- [Cheap2Rich: A Multi-Fidelity Framework for Data Assimilation and System Identification of Multiscale Physics -- Rotating Detonation Engines](https://mlsanity.com/paper/2601.20295/) (2026)
- [OpenReservoirComputing: GPU-Accelerated Reservoir Computing in JAX](https://mlsanity.com/paper/2603.14802/) (2026)
- [EML Trees Are Universal Approximators](https://mlsanity.com/paper/2606.23179/) (2026)
- [Physics-informed reduced-order modelling with equivariant spectral submanifolds](https://mlsanity.com/paper/2608.04239/) (2026)
- [LAtent Phase Inference from Short time sequences using SHallow REcurrent Decoders (LAPIS-SHRED)](https://mlsanity.com/paper/2604.01216/) (2026)
- [Reliability of Probabilistic Emulation of Physical Systems](https://mlsanity.com/paper/2606.12997/) (2026)
- [OrthoReg: Orthogonal Regularization for Hybrid Symbolic-Neural Dynamical Systems](https://mlsanity.com/paper/2606.19145/) (2026)
- [LSTM Variants for Chaotic Dynamical Systems: An Empirical Study on the Lorenz Attractor](https://mlsanity.com/paper/2606.22662/) (2026)
- [The Smart Buildings Challenge: Learning Foundation Models for Real-World Optimization at Scale](https://mlsanity.com/paper/2610.18511/) (2026)

## Abstract (arXiv)

Machine learning (ML) is transforming modeling and control in the physical, engineering, and biological sciences. However, rapid development has outpaced the creation of standardized, objective benchmarks - leading to weak baselines, reporting bias, and inconsistent evaluations across methods. This undermines reproducibility, misguides resource allocation, and obscures scientific progress. To address this, we propose a Common Task Framework (CTF) for scientific machine learning. The CTF features a curated set of datasets and task-specific metrics spanning forecasting, state reconstruction, and generalization under realistic constraints, including noise and limited data. Inspired by the success of CTFs in fields like natural language processing and computer vision, our framework provides a structured, rigorous foundation for head-to-head evaluation of diverse algorithms. As a first step, we benchmark methods on two canonical nonlinear systems: Kuramoto-Sivashinsky and Lorenz. These results illustrate the utility of the CTF in revealing method strengths, limitations, and suitability for specific classes of problems and diverse objectives. Next, we are launching a competition around a global real world sea surface temperature dataset with a true holdout dataset to foster community engagement. Our long-term vision is to replace ad hoc comparisons with standardized evaluations on hidden test sets that raise the bar for rigor and reproducibility in scientific ML.