# Clever Materials: When Models Identify Good Materials for the Wrong Reasons

arXiv: https://arxiv.org/abs/2602.17730 · HTML: https://mlsanity.com/paper/2602.17730/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 60 references that are in the map.
- First posted on arXiv 18 Feb 2026.

## Builds on (most cited first, 15 of 15)

- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [Kosmos: An AI Scientist for Autonomous Discovery](https://mlsanity.com/paper/2511.02824/) (2025)
- [Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification](https://mlsanity.com/paper/1909.01940/) (2019)
- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [How to avoid machine learning pitfalls: a guide for academic researchers](https://mlsanity.com/paper/2108.02497/) (2021)
- [Aviary: training language agents on challenging scientific tasks](https://mlsanity.com/paper/2412.21154/) (2024)
- [Using Machine Learning and Data Mining to Leverage Community Knowledge for the Engineering of Stable Metal-Organic Frameworks](https://mlsanity.com/paper/2106.13327/) (2021)
- [Scaffold Splits Overestimate Virtual Screening Performance](https://mlsanity.com/paper/2406.00873/) (2024)
- [General-Purpose Models for the Chemical Sciences: LLMs and Beyond](https://mlsanity.com/paper/2507.07456/) (2025)
- [Lessons from the trenches on evaluating machine-learning systems in materials science](https://mlsanity.com/paper/2503.10837/) (2025)
- [An autonomous living database for perovskite photovoltaics](https://mlsanity.com/paper/2601.17807/) (2026)
- [Random projections and Kernelised Leave One Cluster Out Cross-Validation: Universal baselines and evaluation tools for supervised machine learning for materials properties](https://mlsanity.com/paper/2206.08841/) (2022)
- [Clever Hans in Chemistry: Chemist Style Signals Confound Activity Prediction on Public Benchmarks](https://mlsanity.com/paper/2512.20924/) (2025)

## Built on by (most cited first, 3 of 3)

- [Reducing cross-sample prediction churn in scientific machine learning](https://mlsanity.com/paper/2605.13826/) (2026)
- [Provenance-grouped protocol for evaluating machine learning in perovskite photovoltaics](https://mlsanity.com/paper/2610.12766/) (2026)
- [Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction](https://mlsanity.com/paper/2610.19906/) (2026)

## Abstract (arXiv)

Machine learning can accelerate materials discovery. Models perform impressively on many benchmarks. However, strong benchmark performance does not imply that a model learned chemistry. I test a concrete alternative hypothesis: that property prediction can be driven by bibliographic confounding. Across five tasks spanning MOFs (thermal and solvent stability), perovskite solar cells (efficiency), batteries (capacity), and TADF emitters (emission wavelength), models trained on standard chemical descriptors predict author, journal, and publication year well above chance. When these predicted metadata ("bibliographic fingerprints") are used as the sole input to a second model, performance is sometimes competitive with conventional descriptor-based predictors. These results show that many datasets do not rule out non-chemical explanations of success. Progress requires routine falsification tests (e.g., group/time splits and metadata ablations), datasets designed to resist spurious correlations, and explicit separation of two goals: predictive utility versus evidence of chemical understanding.