# POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimization

arXiv: https://arxiv.org/abs/2509.21737 · HTML: https://mlsanity.com/paper/2509.21737/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2504.20073/) (2025)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Molecular De Novo Design through Deep Reinforcement Learning](https://mlsanity.com/paper/1704.07555/) (2017)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [Deep Reinforcement Learning for De-Novo Drug Design](https://mlsanity.com/paper/1711.10907/) (2017)
- [Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?](https://mlsanity.com/paper/2402.18272/) (2024)
- [Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization](https://mlsanity.com/paper/2206.12411/) (2022)
- [LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset](https://mlsanity.com/paper/2402.09391/) (2024)
- [ChemLLM: A Chemical Large Language Model](https://mlsanity.com/paper/2402.06852/) (2024)
- [BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations](https://mlsanity.com/paper/2310.07276/) (2023)
- [LiPO: Listwise Preference Optimization through Learning-to-Rank](https://mlsanity.com/paper/2402.01878/) (2024)
- [ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendations](https://mlsanity.com/paper/1908.01425/) (2019)
- [Multi-turn Reinforcement Learning from Preference Human Feedback](https://mlsanity.com/paper/2405.14655/) (2024)
- [Efficient Evolutionary Search Over Chemical Space with Large Language Models](https://mlsanity.com/paper/2406.16976/) (2024)
- [DrugAssist: A Large Language Model for Molecule Optimization](https://mlsanity.com/paper/2401.10334/) (2023)
- [Optimizing Molecules using Efficient Queries from Property Evaluations](https://mlsanity.com/paper/2011.01921/) (2020)
- [A Deep Generative Model for Molecule Optimization via One Fragment Modification](https://mlsanity.com/paper/2012.04231/) (2020)
- [GeLLMO: Generalizing Large Language Models for Multi-property Molecule Optimization](https://mlsanity.com/paper/2502.13398/) (2025)
- [Property Enhanced Instruction Tuning for Multi-task Molecule Generation with Large Language Models](https://mlsanity.com/paper/2412.18084/) (2024)

## Built on by (most cited first, 11 of 11)

- [Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective](https://mlsanity.com/paper/2510.10150/) (2025)
- [TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization](https://mlsanity.com/paper/2601.22776/) (2026)
- [MolMem: Memory-Augmented Agentic Reinforcement Learning for Sample-Efficient Molecular Optimization](https://mlsanity.com/paper/2604.12237/) (2026)
- [CoAct: Co-Active LLM Preference Learning with Human-AI Synergy](https://mlsanity.com/paper/2604.17501/) (2026)
- [GlassMol: Interpretable Molecular Property Prediction with Concept Bottleneck Models](https://mlsanity.com/paper/2603.01274/) (2026)
- [Can AI Scientist Agents Learn from Lab-in-the-Loop Feedback? Evidence from Iterative Perturbation Discovery](https://mlsanity.com/paper/2603.26177/) (2026)
- [FRAGMENTA: End-to-end Fragmentation-based Generative Model with Agentic Tuning for Drug Lead Optimization](https://mlsanity.com/paper/2511.20510/) (2025)
- [Deciphering Scientific Reasoning Steps from Outcome Data for Molecule Optimization](https://mlsanity.com/paper/2603.20262/) (2026)
- [ReContraster: Making Your Posters Stand Out with Regional Contrast](https://mlsanity.com/paper/2604.10442/) (2026)
- [HopRank: Self-Supervised LLM Preference-Tuning on Graphs for Few-Shot Node Classification](https://mlsanity.com/paper/2604.17271/) (2026)
- [Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery](https://mlsanity.com/paper/2610.18449/) (2026)

## Abstract (arXiv)

Lead optimization in drug discovery requires efficiently navigating vast chemical space through iterative cycles to enhance molecular properties while preserving structural similarity to the original lead compound. Despite recent advances, traditional optimization methods struggle with sample efficiency-achieving good optimization performance with limited oracle evaluations. Large Language Models (LLMs) provide a promising approach through their in-context learning and instruction following capabilities, which align naturally with these iterative processes. However, existing LLM-based methods fail to leverage this strength, treating each optimization step independently. To address this, we present POLO (Preference-guided multi-turn Optimization for Lead Optimization), which enables LLMs to learn from complete optimization trajectories rather than isolated steps. At its core, POLO introduces Preference-Guided Policy Optimization (PGPO), a novel reinforcement learning algorithm that extracts learning signals at two complementary levels: trajectory-level optimization reinforces successful strategies, while turn-level preference learning provides dense comparative feedback by ranking intermediate molecules within each trajectory. Through this dual-level learning from intermediate evaluation, POLO achieves superior sample efficiency by fully exploiting each costly oracle call. Extensive experiments demonstrate that POLO achieves 84% average success rate on single-property tasks (2.3x better than baselines) and 50% on multi-property tasks using only 500 oracle evaluations, significantly advancing the state-of-the-art in sample-efficient molecular optimization.