# Calibration and generalizability of probabilistic models on low-data chemical datasets with DIONYSUS

arXiv: https://arxiv.org/abs/2212.01574 · HTML: https://mlsanity.com/paper/2212.01574/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 96 references that are in the map.
- First posted on arXiv 3 Dec 2022.

## Builds on (most cited first, 30 of 35)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Robotics](https://mlsanity.com/paper/1602.04450/) (2016)
- [Variational Bayesian Optimal Experimental Design](https://mlsanity.com/paper/1903.05480/) (2019)
- [NGBoost: Natural Gradient Boosting for Probabilistic Prediction](https://mlsanity.com/paper/1910.03225/) (2019)
- [Low Data Drug Discovery with One-shot Learning](https://mlsanity.com/paper/1611.03199/) (2016)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [Uncertainty Quantification Using Neural Networks for Molecular Property Prediction](https://mlsanity.com/paper/2005.10036/) (2020)
- [Accelerating high-throughput virtual screening through molecular pool-based active learning](https://mlsanity.com/paper/2012.07127/) (2020)
- [A Framework for Interdomain and Multioutput Gaussian Processes](https://mlsanity.com/paper/2003.01115/) (2020)
- [A Network Perspective on Stratification of Multi-Label Data](https://mlsanity.com/paper/1704.08756/) (2017)
- [Calibrated Reliable Regression using Maximum Mean Discrepancy](https://mlsanity.com/paper/2006.10255/) (2020)
- [Gryffin: An algorithm for Bayesian optimization of categorical variables informed by expert knowledge](https://mlsanity.com/paper/2003.12127/) (2020)
- [Machine Learning for Scent: Learning Generalizable Perceptual Representations of Small Molecules](https://mlsanity.com/paper/1910.10685/) (2019)
- [Reliable Graph Neural Networks for Drug Discovery Under Distributional Shift](https://mlsanity.com/paper/2111.12951/) (2021)
- [Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning](https://mlsanity.com/paper/1902.00925/) (2019)
- [Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networks](https://mlsanity.com/paper/2107.06068/) (2021)

## Built on by (most cited first, 10 of 10)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [El Agente: An Autonomous Agent for Quantum Chemistry](https://mlsanity.com/paper/2505.02484/) (2025)
- [GAUCHE: A Library for Gaussian Processes in Chemistry](https://mlsanity.com/paper/2212.04450/) (2022)
- [Improving Self-supervised Molecular Representation Learning using Persistent Homology](https://mlsanity.com/paper/2311.17327/) (2023)
- [The fast committor machine: Interpretable prediction with kernels](https://mlsanity.com/paper/2405.10410/) (2024)
- [Ranking over Regression for Bayesian Optimization and Molecule Selection](https://mlsanity.com/paper/2410.09290/) (2024)
- [DOMINO++: Domain-aware Loss Regularization for Deep Learning Generalizability](https://mlsanity.com/paper/2308.10453/) (2023)
- [Frugal Bayesian Optimization: Scalable Surrogates for Data- and Resource-Limited Discovery](https://mlsanity.com/paper/2607.29225/) (2026)
- [Applications of Gaussian Processes at Extreme Lengthscales: From Molecules to Black Holes](https://mlsanity.com/paper/2303.14291/) (2023)
- [Benchmarking Peptide-Protein Affinity Prediction Across Peptide and Target Shifts](https://mlsanity.com/paper/2608.30175/) (2026)

## Abstract (arXiv)

Deep learning models that leverage large datasets are often the state of the art for modelling molecular properties. When the datasets are smaller (< 2000 molecules), it is not clear that deep learning approaches are the right modelling tool. In this work we perform an extensive study of the calibration and generalizability of probabilistic machine learning models on small chemical datasets. Using different molecular representations and models, we analyse the quality of their predictions and uncertainties in a variety of tasks (binary, regression) and datasets. We also introduce two simulated experiments that evaluate their performance: (1) Bayesian optimization guided molecular design, (2) inference on out-of-distribution data via ablated cluster splits. We offer practical insights into model and feature choice for modelling small chemical datasets, a common scenario in new chemical experiments. We have packaged our analysis into the DIONYSUS repository, which is open sourced to aid in reproducibility and extension to new datasets.