# Learning To Navigate The Synthetically Accessible Chemical Space Using Reinforcement Learning

arXiv: https://arxiv.org/abs/2004.12485 · HTML: https://mlsanity.com/paper/2004.12485/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2020.

## Built on by (most cited first, 30 of 57)

- [Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation](https://mlsanity.com/paper/2106.04399/) (2021)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation](https://mlsanity.com/paper/2110.01219/) (2021)
- [Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching](https://mlsanity.com/paper/2206.13602/) (2022)
- [Differentiable Scaffolding Tree for Molecular Optimization](https://mlsanity.com/paper/2109.10469/) (2021)
- [Amortized Tree Generation for Bottom-up Synthesis Planning and Synthesizable Molecular Design](https://mlsanity.com/paper/2110.06389/) (2021)
- [RGFN: Synthesizable Molecular Generation Using GFlowNets](https://mlsanity.com/paper/2406.08506/) (2024)
- [JANUS: Parallel Tempered Genetic Algorithm Guided by Deep Neural Networks for Inverse Molecular Design](https://mlsanity.com/paper/2106.04011/) (2021)
- [ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedback](https://mlsanity.com/paper/2305.18090/) (2023)
- [SynFlowNet: Design of Diverse and Novel Molecules with Synthesis Constraints](https://mlsanity.com/paper/2405.01155/) (2024)
- [Double-Ended Synthesis Planning with Goal-Constrained Bidirectional Search](https://mlsanity.com/paper/2407.06334/) (2024)
- [Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materials](https://mlsanity.com/paper/2306.09375/) (2023)
- [Generative Flows on Synthetic Pathway for Drug Design](https://mlsanity.com/paper/2410.04542/) (2024)
- [GeneDisco: A Benchmark for Experimental Design in Drug Discovery](https://mlsanity.com/paper/2110.11875/) (2021)
- [Generative Artificial Intelligence for Navigating Synthesizable Chemical Space](https://mlsanity.com/paper/2410.03494/) (2024)
- [Retro-fallback: retrosynthetic planning in an uncertain world](https://mlsanity.com/paper/2310.09270/) (2023)
- [Retrosynthetic Planning with Experience-Guided Monte Carlo Tree Search](https://mlsanity.com/paper/2112.06028/) (2021)
- [Multi-domain Distribution Learning for De Novo Drug Design](https://mlsanity.com/paper/2508.17815/) (2025)
- [A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Science](https://mlsanity.com/paper/2403.12636/) (2024)
- [Modern Hopfield Networks for Few- and Zero-Shot Reaction Template Prediction](https://mlsanity.com/paper/2104.03279/) (2021)
- [LLM-Augmented Chemical Synthesis and Design Decision Programs](https://mlsanity.com/paper/2505.07027/) (2025)
- [Unifying Likelihood-free Inference with Black-box Optimization and Beyond](https://mlsanity.com/paper/2110.03372/) (2021)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Conditional β-VAE for De Novo Molecular Generation](https://mlsanity.com/paper/2205.01592/) (2022)
- [Molecule Design by Latent Space Energy-Based Modeling and Gradual Distribution Shifting](https://mlsanity.com/paper/2306.14902/) (2023)
- [Utilizing Reinforcement Learning for de novo Drug Design](https://mlsanity.com/paper/2303.17615/) (2023)
- [Entropy-Reinforced Planning with Large Language Models for Drug Discovery](https://mlsanity.com/paper/2406.07025/) (2024)
- [Challenging reaction prediction models to generalize to novel chemistry](https://mlsanity.com/paper/2501.06669/) (2025)
- [DrugImproverGPT: A Large Language Model for Drug Optimization with Fine-Tuning via Structured Policy Optimization](https://mlsanity.com/paper/2502.07237/) (2025)
- [Evaluating Molecule Synthesizability via Retrosynthetic Planning and Reaction Prediction](https://mlsanity.com/paper/2411.08306/) (2024)

## Abstract (arXiv)

Over the last decade, there has been significant progress in the field of machine learning for de novo drug design, particularly in deep generative models. However, current generative approaches exhibit a significant challenge as they do not ensure that the proposed molecular structures can be feasibly synthesized nor do they provide the synthesis routes of the proposed small molecules, thereby seriously limiting their practical applicability. In this work, we propose a novel forward synthesis framework powered by reinforcement learning (RL) for de novo drug design, Policy Gradient for Forward Synthesis (PGFS), that addresses this challenge by embedding the concept of synthetic accessibility directly into the de novo drug design system. In this setup, the agent learns to navigate through the immense synthetically accessible chemical space by subjecting commercially available small molecule building blocks to valid chemical reactions at every time step of the iterative virtual multi-step synthesis process. The proposed environment for drug discovery provides a highly challenging test-bed for RL algorithms owing to the large state space and high-dimensional continuous action space with hierarchical actions. PGFS achieves state-of-the-art performance in generating structures with high QED and penalized clogP. Moreover, we validate PGFS in an in-silico proof-of-concept associated with three HIV targets. Finally, we describe how the end-to-end training conceptualized in this study represents an important paradigm in radically expanding the synthesizable chemical space and automating the drug discovery process.