# Uncertainty-aware reinforcement learning for chemical language models

arXiv: https://arxiv.org/abs/2606.24990 · HTML: https://mlsanity.com/paper/2606.24990/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 49 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 13 of 13)

- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [Reinforcement learning](https://mlsanity.com/paper/2405.10369/) (2024)
- [Optimization of Molecules via Deep Reinforcement Learning](https://mlsanity.com/paper/1810.08678/) (2018)
- [Exploring Chemical Space with Score-based Out-of-distribution Generation](https://mlsanity.com/paper/2206.07632/) (2022)
- [Uncertainty Quantification Using Neural Networks for Molecular Property Prediction](https://mlsanity.com/paper/2005.10036/) (2020)
- [A Review of Uncertainty for Deep Reinforcement Learning](https://mlsanity.com/paper/2208.09052/) (2022)
- [Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures](https://mlsanity.com/paper/2010.03039/) (2020)
- [Exploring applications of deep reinforcement learning for real-world autonomous driving systems](https://mlsanity.com/paper/1901.01536/) (2019)
- [ACEGEN: Reinforcement learning of generative chemical agents for drug discovery](https://mlsanity.com/paper/2405.04657/) (2024)
- [A General Approach for Determining Applicability Domain of Machine Learning Models](https://mlsanity.com/paper/2406.05143/) (2024)

## Abstract (arXiv)

Reinforcement Learning (RL) has become a powerful paradigm for de novo molecular design, enabling Chemical Language Models (CLMs) to navigate and explore the chemical space while optimizing specific desired properties. However, the existing RL frameworks treat all scoring functions as deterministic oracles, neglecting the inherent uncertainty attached to the predictions of the different molecular properties. This can lead to the exploration of highly-uncertain regions of the chemical space, focusing on the generation of highly scored molecules which are poorly supported by the training data. This can destabilize the optimization process, yielding predictions that are far from their true values. We propose and compare two complementary ways of incorporating predictive uncertainty into RL. In the first one, uncertainty is treated as an additional optimization objective and incorporated along with the rest of the scoring functions, allowing the policy to trade off exploitation against reliability. Secondly, uncertainty is used to modulate policy updates, reducing the influence of molecules whose properties lie far outside the scoring function confidence domain. Both approaches were evaluated across three different settings: (i) a controlled model system, in which the prediction error is modeled as a Gaussian distribution, with a variance proportional to the distance to the training data; and two real-world tasks, making use of (ii) ChemProp models and (iii) a Conformal Prediction wrapper applied to a Random forest classifier. We show that uncertainty-aware RL enables CLMs to explore chemical space more robustly by favoring lower-uncertainty regions. This leads to more reliable hit discovery without compromising molecular score, increasing the true hit rate by 0.25 (from 0.5 to 0.75), and nearly doubling the total number of true hits.