# Searching for High-Value Molecules Using Reinforcement Learning and Transformers

arXiv: https://arxiv.org/abs/2310.02902 · HTML: https://mlsanity.com/paper/2310.02902/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 82 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation](https://mlsanity.com/paper/2106.04399/) (2021)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation](https://mlsanity.com/paper/1806.02473/) (2018)
- [GuacaMol: Benchmarking Models for De Novo Molecular Design](https://mlsanity.com/paper/1811.09621/) (2018)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Optimization of Molecules via Deep Reinforcement Learning](https://mlsanity.com/paper/1810.08678/) (2018)
- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization](https://mlsanity.com/paper/2206.12411/) (2022)
- [Robustness of Graph Neural Networks at Scale](https://mlsanity.com/paper/2110.14038/) (2021)
- [Exploring Chemical Space with Score-based Out-of-distribution Generation](https://mlsanity.com/paper/2206.07632/) (2022)
- [Molecular representation learning with language models and domain-relevant auxiliary tasks](https://mlsanity.com/paper/2011.13230/) (2020)
- [SELFIES and the future of molecular string representations](https://mlsanity.com/paper/2204.00056/) (2022)
- [Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes](https://mlsanity.com/paper/2211.15144/) (2022)
- [Recurrent Neural Networks (RNNs): A gentle Introduction and Overview](https://mlsanity.com/paper/1912.05911/) (2019)

## Built on by (most cited first, 18 of 18)

- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL](https://mlsanity.com/paper/2410.20254/) (2024)
- [Less can be more for predicting properties with large language models](https://mlsanity.com/paper/2406.17295/) (2024)
- [Can LLMs Generate Diverse Molecules? Towards Alignment with Structural Diversity](https://mlsanity.com/paper/2410.03138/) (2024)
- [AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learning](https://mlsanity.com/paper/2310.15719/) (2023)
- [Regulatory DNA sequence Design with Reinforcement Learning](https://mlsanity.com/paper/2503.07981/) (2025)
- [CrystalGym: A New Benchmark for Materials Discovery Using Reinforcement Learning](https://mlsanity.com/paper/2509.23156/) (2025)
- [Any-Property-Conditional Molecule Generation with Self-Criticism using Spanning Trees](https://mlsanity.com/paper/2407.09357/) (2024)
- [Mol-MoE: Training Preference-Guided Routers for Molecule Generation](https://mlsanity.com/paper/2502.05633/) (2025)
- [Leveraging Partial SMILES Validation Scheme for Enhanced Drug Design in Reinforcement Learning Frameworks](https://mlsanity.com/paper/2505.00530/) (2025)
- [NPGPT: Natural Product-Like Compound Generation with GPT-based Chemical Language Models](https://mlsanity.com/paper/2411.12886/) (2024)
- [A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation](https://mlsanity.com/paper/2503.05696/) (2025)
- [FREED++: Improving RL Agents for Fragment-Based Molecule Generation by Thorough Reproduction](https://mlsanity.com/paper/2401.09840/) (2024)
- [TransDiffSBDD: Causality-Aware Multi-Modal Structure-Based Drug Design](https://mlsanity.com/paper/2503.20913/) (2025)
- [Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning](https://mlsanity.com/paper/2510.25311/) (2025)
- [How well can off-the-shelf LLMs elucidate molecular structures from mass spectra using chain-of-thought reasoning?](https://mlsanity.com/paper/2601.06289/) (2026)
- [MolLIBRA: Genetic Molecular Optimization with Multi-Fingerprint Surrogates and Text-Molecule Aligned Critic](https://mlsanity.com/paper/2602.07002/) (2026)
- [C-MORAL: Controllable Multi-Objective Molecular Optimization with Reinforcement Alignment for LLMs](https://mlsanity.com/paper/2604.23061/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) over text representations can be effective for finding high-value policies that can search over graphs. However, RL requires careful structuring of the search space and algorithm design to be effective in this challenge. Through extensive experiments, we explore how different design choices for text grammar and algorithmic choices for training can affect an RL policy's ability to generate molecules with desired properties. We arrive at a new RL-based molecular design algorithm (ChemRLformer) and perform a thorough analysis using 25 molecule design tasks, including computationally complex protein docking simulations. From this analysis, we discover unique insights in this problem space and show that ChemRLformer achieves state-of-the-art performance while being more straightforward than prior work by demystifying which design choices are actually helpful for text-based molecule design.