# Preference Optimization for Molecular Language Models

arXiv: https://arxiv.org/abs/2310.12304 · HTML: https://mlsanity.com/paper/2310.12304/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Preference optimization for molecular design, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 4 of its 11 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference optimization for molecular design](https://mlsanity.com/alignment/preference-optimization-for-molecular-design/)

## Builds on (most cited first, 4 of 4)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models](https://mlsanity.com/paper/1811.12823/) (2018)
- [ChemTS: An Efficient Python Library for de novo Molecular Generation](https://mlsanity.com/paper/1710.00616/) (2017)
- [Generating Focussed Molecule Libraries for Drug Discovery with Recurrent Neural Networks](https://mlsanity.com/paper/1701.01329/) (2017)

## Built on by (most cited first, 7 of 7)

- [Multi-domain Distribution Learning for De Novo Drug Design](https://mlsanity.com/paper/2508.17815/) (2025)
- [SmileyLlama: Modifying Large Language Models for Directed Chemical Space Exploration](https://mlsanity.com/paper/2409.02231/) (2024)
- [Aligning Transformers with Continuous Feedback via Energy Rank Alignment](https://mlsanity.com/paper/2405.12961/) (2024)
- [Boltzmann-Aligned Inverse Folding Model as a Predictor of Mutational Effects on Protein-Protein Interactions](https://mlsanity.com/paper/2410.09543/) (2024)
- [Improving Inverse Folding for Peptide Design with Diversity-regularized Direct Preference Optimization](https://mlsanity.com/paper/2410.19471/) (2024)
- [Preference optimization of protein language models as a multi-objective binder design paradigm](https://mlsanity.com/paper/2403.04187/) (2024)
- [LLM-to-Flow Distillation: Teaching Natural Language Goals to Scientific Generators](https://mlsanity.com/paper/2610.17246/) (2026)

## Abstract (arXiv)

Molecular language modeling is an effective approach to generating novel chemical structures. However, these models do not a priori encode certain preferences a chemist may desire. We investigate the use of fine-tuning using Direct Preference Optimization to better align generated molecules with chemist preferences. Our findings suggest that this approach is simple, efficient, and highly effective.