# Round-trip Reinforcement Learning: Self-Consistent Training for Better Chemical LLMs

arXiv: https://arxiv.org/abs/2510.01527 · HTML: https://mlsanity.com/paper/2510.01527/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 59 references that are in the map.
- First posted on arXiv 1 Oct 2025.

## Builds on (most cited first, 30 of 44)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://mlsanity.com/paper/2501.04519/) (2025)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [R-Zero: Self-Evolving Reasoning LLM from Zero Data](https://mlsanity.com/paper/2508.05004/) (2025)
- [Spurious Rewards: Rethinking Training Signals in RLVR](https://mlsanity.com/paper/2506.10947/) (2025)
- [Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discovery](https://mlsanity.com/paper/1803.09518/) (2018)
- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning](https://mlsanity.com/paper/2505.15134/) (2025)
- [Synthetic QA Corpora Generation with Roundtrip Consistency](https://mlsanity.com/paper/1906.05416/) (2019)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset](https://mlsanity.com/paper/2402.09391/) (2024)
- [ChemLLM: A Chemical Large Language Model](https://mlsanity.com/paper/2402.06852/) (2024)
- [Predicting Organic Reaction Outcomes with Weisfeiler-Lehman Network](https://mlsanity.com/paper/1709.04555/) (2017)
- [InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery](https://mlsanity.com/paper/2311.16208/) (2023)
- [RLPR: Extrapolating RLVR to General Domains without Verifiers](https://mlsanity.com/paper/2506.18254/) (2025)
- [DrugAssist: A Large Language Model for Molecule Optimization](https://mlsanity.com/paper/2401.10334/) (2023)
- [BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning](https://mlsanity.com/paper/2402.17810/) (2024)
- [Developing ChemDFM as a large language foundation model for chemistry](https://mlsanity.com/paper/2401.14818/) (2024)
- [Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.13445/) (2025)
- [No Free Lunch: Rethinking Internal Feedback for LLM Reasoning](https://mlsanity.com/paper/2506.17219/) (2025)
- [Nature Language Model: Deciphering the Language of Nature for Scientific Discovery](https://mlsanity.com/paper/2502.07527/) (2025)
- [Dual Supervised Learning](https://mlsanity.com/paper/1707.00415/) (2017)
- [MolecularGPT: Open Large Language Model (LLM) for Few-Shot Molecular Property Prediction](https://mlsanity.com/paper/2406.12950/) (2024)
- [Unsupervised Evaluation of Code LLMs with Round-Trip Correctness](https://mlsanity.com/paper/2402.08699/) (2024)
- [UniMoT: Unified Molecule-Text Language Model with Discrete Token Representation](https://mlsanity.com/paper/2408.00863/) (2024)
- [L+M-24: Building a Dataset for Language + Molecules @ ACL 2024](https://mlsanity.com/paper/2403.00791/) (2024)

## Built on by (most cited first, 2 of 2)

- [The Communication Bottleneck: A Round-Trip Study of Tree-Structured Expression Serialization in Language Models](https://mlsanity.com/paper/2609.21509/) (2026)
- [Single-step Retrosynthesis via Reinforcement Learning from Forward Feedback](https://mlsanity.com/paper/2610.15141/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are emerging as versatile foundation models for computational chemistry, handling bidirectional tasks like reaction prediction and retrosynthesis. However, these models often lack round-trip consistency. For instance, a state-of-the-art chemical LLM may successfully caption a molecule, yet be unable to accurately reconstruct the original structure from its own generated text. This inconsistency suggests that models are learning unidirectional memorization rather than flexible mastery. Indeed, recent work has demonstrated a strong correlation between a model's round-trip consistency and its performance on the primary tasks. This strong correlation reframes consistency into a direct target for model improvement. We therefore introduce Round-Trip Reinforcement Learning (RTRL), a novel framework that trains a model to improve its consistency by using the success of a round-trip transformation as a reward signal. We further propose an iterative variant where forward and reverse mappings alternately train each other in a self-improvement loop, a process that is highly data-efficient and notably effective with the massive amount of unlabelled data common in chemistry. Experiments demonstrate that RTRL significantly boosts performance and consistency over strong baselines across supervised, self-supervised, and synthetic data regimes. This work shows that round-trip consistency is not just a desirable property but a trainable objective, offering a new path toward more robust and reliable foundation models.