# Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation

arXiv: https://arxiv.org/abs/2110.01219 · HTML: https://mlsanity.com/paper/2110.01219/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 64 references that are in the map.
- First posted on arXiv 4 Oct 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [MolGAN: An implicit generative model for small molecular graphs](https://mlsanity.com/paper/1805.11973/) (2018)
- [Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation](https://mlsanity.com/paper/1806.02473/) (2018)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Hierarchical Generation of Molecular Graphs using Structural Motifs](https://mlsanity.com/paper/2002.03230/) (2020)
- [Optimization of Molecules via Deep Reinforcement Learning](https://mlsanity.com/paper/1810.08678/) (2018)
- [Soft Actor-Critic for Discrete Action Settings](https://mlsanity.com/paper/1910.07207/) (2019)
- [Deep Reinforcement Learning for De-Novo Drug Design](https://mlsanity.com/paper/1711.10907/) (2017)
- [Self-Imitation Learning](https://mlsanity.com/paper/1806.05635/) (2018)
- [Learning to Extend Molecular Scaffolds with Structural Motifs](https://mlsanity.com/paper/2103.03864/) (2021)
- [Learning To Navigate The Synthetically Accessible Chemical Space Using Reinforcement Learning](https://mlsanity.com/paper/2004.12485/) (2020)
- [Making Efficient Use of Demonstrations to Solve Hard Exploration Problems](https://mlsanity.com/paper/1909.01387/) (2019)
- [Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment](https://mlsanity.com/paper/1908.02388/) (2019)
- [Curiosity in exploring chemical space: Intrinsic rewards for deep molecular reinforcement learning](https://mlsanity.com/paper/2012.11293/) (2020)
- [Self-Imitation Advantage Learning](https://mlsanity.com/paper/2012.11989/) (2020)
- [Predictive PER: Balancing Priority and Diversity towards Stable Deep Reinforcement Learning](https://mlsanity.com/paper/2011.13093/) (2020)

## Built on by (most cited first, 30 of 68)

- [Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding](https://mlsanity.com/paper/2408.08252/) (2024)
- [Exploring Chemical Space with Score-based Out-of-distribution Generation](https://mlsanity.com/paper/2206.07632/) (2022)
- [MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design](https://mlsanity.com/paper/2203.14500/) (2022)
- [GenMol: A Drug Discovery Generalist with Discrete Diffusion](https://mlsanity.com/paper/2501.06158/) (2025)
- [Reinforced Genetic Algorithm for Structure-based Drug Design](https://mlsanity.com/paper/2211.16508/) (2022)
- [De Novo Molecular Generation via Connection-aware Motif Mining](https://mlsanity.com/paper/2302.01129/) (2023)
- [Dynamic Search for Inference-Time Alignment in Diffusion Models](https://mlsanity.com/paper/2503.02039/) (2025)
- [Group SELFIES: A Robust Fragment-Based Molecular String Representation](https://mlsanity.com/paper/2211.13322/) (2022)
- [Coarse-to-Fine: a Hierarchical Diffusion Model for Molecule Generation in 3D](https://mlsanity.com/paper/2305.13266/) (2023)
- [Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Models](https://mlsanity.com/paper/2502.11420/) (2025)
- [Molecule Generation with Fragment Retrieval Augmentation](https://mlsanity.com/paper/2411.12078/) (2024)
- [Drug Discovery with Dynamic Goal-aware Fragments](https://mlsanity.com/paper/2310.00841/) (2023)
- [De novo Drug Design using Reinforcement Learning with Multiple GPT Agents](https://mlsanity.com/paper/2401.06155/) (2023)
- [Searching for High-Value Molecules Using Reinforcement Learning and Transformers](https://mlsanity.com/paper/2310.02902/) (2023)
- [GeLLMO: Generalizing Large Language Models for Multi-property Molecule Optimization](https://mlsanity.com/paper/2502.13398/) (2025)
- [Saturn: Sample-efficient Generative Molecular Design using Memory Manipulation](https://mlsanity.com/paper/2405.17066/) (2024)
- [MatSciML: A Broad, Multi-Task Benchmark for Solid-State Materials Modeling](https://mlsanity.com/paper/2309.05934/) (2023)
- [Molecule Generation for Drug Design: a Graph Learning Perspective](https://mlsanity.com/paper/2202.09212/) (2022)
- [Graph Diffusion Policy Optimization](https://mlsanity.com/paper/2402.16302/) (2024)
- [Text-guided multi-property molecular optimization with a diffusion language model](https://mlsanity.com/paper/2410.13597/) (2024)
- [Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Design](https://mlsanity.com/paper/2507.00445/) (2025)
- [Deep Learning Methods for Small Molecule Drug Discovery: A Survey](https://mlsanity.com/paper/2303.00313/) (2023)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [SynLlama: Generating Synthesizable Molecules and Their Analogs with Large Language Models](https://mlsanity.com/paper/2503.12602/) (2025)
- [Utilizing Reinforcement Learning for de novo Drug Design](https://mlsanity.com/paper/2303.17615/) (2023)
- [MAGNet: Motif-Agnostic Generation of Molecules from Shapes](https://mlsanity.com/paper/2305.19303/) (2023)
- [Mol-AIR: Molecular Reinforcement Learning with Adaptive Intrinsic Rewards for Goal-directed Molecular Generation](https://mlsanity.com/paper/2403.20109/) (2024)
- [MAGE: Model-Level Graph Neural Networks Explanations via Motif-based Graph Generation](https://mlsanity.com/paper/2405.12519/) (2024)
- [FragFM: Hierarchical Framework for Efficient Molecule Generation via Fragment-Level Discrete Flow Matching](https://mlsanity.com/paper/2502.15805/) (2025)
- [The power of motifs as inductive bias for learning molecular distributions](https://mlsanity.com/paper/2306.17246/) (2023)

## Abstract (arXiv)

Recently, utilizing reinforcement learning (RL) to generate molecules with desired properties has been highlighted as a promising strategy for drug design. A molecular docking program - a physical simulation that estimates protein-small molecule binding affinity - can be an ideal reward scoring function for RL, as it is a straightforward proxy of the therapeutic potential. Still, two imminent challenges exist for this task. First, the models often fail to generate chemically realistic and pharmacochemically acceptable molecules. Second, the docking score optimization is a difficult exploration problem that involves many local optima and less smooth surfaces with respect to molecular structure. To tackle these challenges, we propose a novel RL framework that generates pharmacochemically acceptable molecules with large docking scores. Our method - Fragment-based generative RL with Explorative Experience replay for Drug design (FREED) - constrains the generated molecules to a realistic and qualified chemical space and effectively explores the space to find drugs by coupling our fragment-based generation method and a novel error-prioritized experience replay (PER). We also show that our model performs well on both de novo and scaffold-based schemes. Our model produces molecules of higher quality compared to existing methods while achieving state-of-the-art performance on two of three targets in terms of the docking scores of the generated molecules. We further show with ablation studies that our method, predictive error-PER (FREED(PE)), significantly improves the model performance.