# Single-step Retrosynthesis via Reinforcement Learning from Forward Feedback

arXiv: https://arxiv.org/abs/2610.15141 · HTML: https://mlsanity.com/paper/2610.15141/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)
- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [Retrosynthesis Prediction with Conditional Graph Logic Network](https://mlsanity.com/paper/2001.01408/) (2020)
- [Molecule Edit Graph Attention Network: Modeling Chemical Reactions as Sequences of Graph Edits](https://mlsanity.com/paper/2006.15426/) (2020)
- [Retrosynthetic reaction prediction using neural sequence-to-sequence models](https://mlsanity.com/paper/1706.01643/) (2017)
- [A Graph to Graphs Framework for Retrosynthesis Prediction](https://mlsanity.com/paper/2003.12725/) (2020)
- [RetroXpert: Decompose Retrosynthesis Prediction like a Chemist](https://mlsanity.com/paper/2011.02893/) (2020)
- [Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction](https://mlsanity.com/paper/2110.09681/) (2021)
- [Learning Graph Models for Retrosynthesis Prediction](https://mlsanity.com/paper/2006.07038/) (2020)
- [Training a Scientific Reasoning Model for Chemistry](https://mlsanity.com/paper/2506.17238/) (2025)
- [Root-aligned SMILES: A Tight Representation for Chemical Reaction Prediction](https://mlsanity.com/paper/2203.11444/) (2022)
- [RetroBridge: Modeling Retrosynthesis with Markov Bridges](https://mlsanity.com/paper/2308.16212/) (2023)
- [Retrosynthesis Prediction with Local Template Retrieval](https://mlsanity.com/paper/2306.04123/) (2023)
- [RetroGFN: Diverse and Feasible Retrosynthesis using GFlowNets](https://mlsanity.com/paper/2406.18739/) (2024)
- [G2GT: Retrosynthesis Prediction with Graph to Graph Attention Neural Network and Self-Training](https://mlsanity.com/paper/2204.08608/) (2022)
- [RCsearcher: Reaction Center Identification in Retrosynthesis via Deep Q-Learning](https://mlsanity.com/paper/2301.12071/) (2023)
- [RETRO SYNFLOW: Discrete Flow Matching for Accurate and Diverse Single-Step Retrosynthesis](https://mlsanity.com/paper/2506.04439/) (2025)
- [RLSynC: Offline-Online Reinforcement Learning for Synthon Completion](https://mlsanity.com/paper/2309.02671/) (2023)
- [Retrosynthesis Prediction via Search in (Hyper) Graph](https://mlsanity.com/paper/2402.06772/) (2024)
- [Round-trip Reinforcement Learning: Self-Consistent Training for Better Chemical LLMs](https://mlsanity.com/paper/2510.01527/) (2025)

## Abstract (arXiv)

Single-step retrosynthesis is a one-to-many inverse problem, yet most models are trained and evaluated against a single recorded reactant set. Reference-based training gives no explicit positive signal for unrecorded alternatives, and exact-match evaluation rejects non-matching predictions regardless of chemical plausibility. We propose Retrosynthesis via Reinforcement Learning from Forward Feedback (RetroRLFF), which optimizes reactant generation using both reference agreement and forward consistency with the target product. Starting from a supervised decoder-only Transformer, RetroRLFF generates complete reactant candidates and uses a frozen forward reaction model to predict products. A sequence-level reward combines reference agreement and forward consistency while penalizing invalid outputs. Direct Advantage-Based Policy Optimization updates the policy without human preference annotations or a preference-trained reward model. On USPTO-50K, RetroRLFF outperforms all 14 baselines in candidate-level round-trip accuracy at k = 1, 3, 5, and 10. With approximately 50 million parameters, it achieves 90.1% top-1 round-trip accuracy, exceeding R-SMILES, the strongest competing baseline on this metric, by 8.5 percentage points. Compared with its supervised policy, round-trip accuracy increases from 83.1% to 90.1%, while exact-match accuracy is maintained at 57.9% versus 57.5%. In an anonymized expert assessment of 131 randomly selected test cases, organic synthesis experts prefer RetroRLFF over EditRetro in 67.9% of comparisons. These results support forward feedback as a training signal for improving forward consistency while retaining top-1 reference recovery. Expert assessment provides complementary evidence of synthetic feasibility on the selected cases, although neither forward consistency nor expert preference establishes experimental feasibility.