# MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery

arXiv: https://arxiv.org/abs/2606.19624 · HTML: https://mlsanity.com/paper/2606.19624/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 59 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 20 of 20)

- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models](https://mlsanity.com/paper/1811.12823/) (2018)
- [Natural Emergent Misalignment from Reward Hacking in Production RL](https://mlsanity.com/paper/2511.18397/) (2025)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [MassSpecGym: A benchmark for the discovery and identification of molecules](https://mlsanity.com/paper/2410.23326/) (2024)
- [The Problem with Metrics is a Fundamental Problem for AI](https://mlsanity.com/paper/2002.08512/) (2020)
- [DiffMS: Diffusion Generation of Molecules Conditioned on Mass Spectra](https://mlsanity.com/paper/2502.09571/) (2025)
- [Unraveling Molecular Structure: A Multimodal Spectroscopic Dataset for Chemistry](https://mlsanity.com/paper/2407.17492/) (2024)
- [Prefix-Tree Decoding for Predicting Mass Spectra from Molecules](https://mlsanity.com/paper/2303.06470/) (2023)
- [MADGEN: Mass-Spec attends to De Novo Molecular generation](https://mlsanity.com/paper/2501.01950/) (2025)
- [MS-BART: Unified Modeling of Mass Spectra and Molecules for Structure Elucidation](https://mlsanity.com/paper/2510.20615/) (2025)
- [JESTR: Joint Embedding Space Technique for Ranking Candidate Molecules for the Annotation of Untargeted Metabolomics Data](https://mlsanity.com/paper/2411.14464/) (2024)
- [Revealing data leakage in protein interaction benchmarks](https://mlsanity.com/paper/2404.10457/) (2024)
- [Test-Time Tuned Language Models Enable End-to-end De Novo Molecular Structure Generation from MS/MS Spectra](https://mlsanity.com/paper/2510.23746/) (2025)
- [Small molecule retrieval from tandem mass spectrometry: what are we optimizing for?](https://mlsanity.com/paper/2602.16507/) (2026)
- [FRIGID: Scaling Diffusion-Based Molecular Generation from Mass Spectra at Training and Inference Time](https://mlsanity.com/paper/2604.16648/) (2026)
- [Comparative Analysis of Formula and Structure Prediction from Tandem Mass Spectra](https://mlsanity.com/paper/2601.00941/) (2026)
- [One Small Step with Fingerprints, One Giant Leap for De Novo Molecule Generation from Mass Spectra](https://mlsanity.com/paper/2508.04180/) (2025)
- [How Evaluation Choices Distort the Outcome of Generative Drug Discovery](https://mlsanity.com/paper/2501.05457/) (2024)

## Built on by (most cited first, 3 of 3)

- [GLACIER: Rethinking Mass Spectrum Prediction as an Object Detection Problem](https://mlsanity.com/paper/2606.29161/) (2026)
- [Nearest-neighbour baselines for fingerprint prediction from MS/MS spectra under different assumptions](https://mlsanity.com/paper/2610.02249/) (2026)
- [Critical Mass: Structure-Disjoint Splits Drift With the Libraries They Are Built On](https://mlsanity.com/paper/2610.12694/) (2026)

## Abstract (arXiv)

Reliable benchmarking is critical for developing machine learning models for tandem mass spectrometry (MS/MS) based molecule discovery. Subtle issues in experimental design and model evaluation procedures can degrade the trustworthiness of such benchmarks and lead to erroneous conclusions. We conduct a thorough review of model evaluation issues in the recent MS/MS machine learning literature, using the standard MassSpecGym benchmark suite as a case study to illustrate the impact of these issues. We find evaluation issues in at least 17 of 26 papers reporting MassSpecGym benchmark results in the first year of its adoption. We isolate three classes of failures: (i) data leakage, (ii) shortcut learning, and (iii) implementation bugs and metric divergence. Through extensive experimentation and code replication, we quantify the impact of these issues and show how they corrupt the evaluation standards MassSpecGym was designed to enforce. We distill our findings into recommendations generalizable to MS/MS challenges, benchmarks, and custom evaluation setups. We also release MassSpecGym v1.5, an implementation of our recommendations in the MassSpecGym benchmarking suite which addresses the failure modes identified in this audit. MassSpecGym v1.5 is publicly available at https://github.com/pluskal-lab/MassSpecGym.