# g-DPO: Scalable Preference Optimization for Protein Language Models

arXiv: https://arxiv.org/abs/2510.19474 · HTML: https://mlsanity.com/paper/2510.19474/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Preference optimization for molecular design, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference optimization for molecular design](https://mlsanity.com/alignment/preference-optimization-for-molecular-design/)

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [Preference Ranking Optimization for Human Alignment](https://mlsanity.com/paper/2306.17492/) (2023)
- [Unlocking Guidance for Discrete State-Space Diffusion and Flow Models](https://mlsanity.com/paper/2406.01572/) (2024)
- [Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences](https://mlsanity.com/paper/2403.07230/) (2024)
- [Curriculum Direct Preference Optimization for Diffusion and Consistency Models](https://mlsanity.com/paper/2405.13637/) (2024)
- [Protein Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.17504/) (2025)
- [DiscoBAX: Discovery of Optimal Intervention Sets in Genomic Experiment Design](https://mlsanity.com/paper/2312.04064/) (2023)
- [ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment](https://mlsanity.com/paper/2505.19241/) (2025)
- [Steering Generative Models with Experimental Data for Protein Fitness Optimization](https://mlsanity.com/paper/2505.15093/) (2025)
- [Improving Protein Sequence Design through Designability Preference Optimization](https://mlsanity.com/paper/2506.00297/) (2025)
- [K-order Ranking Preference Optimization for Large Language Models](https://mlsanity.com/paper/2506.00441/) (2025)
- [Controllable Protein Sequence Generation with LLM Preference Optimization](https://mlsanity.com/paper/2501.15007/) (2025)
- [De Novo Molecular Design Enabled by Direct Preference Optimization and Curriculum Learning](https://mlsanity.com/paper/2504.01389/) (2025)
- [Protein Inverse Folding From Structure Feedback](https://mlsanity.com/paper/2506.03028/) (2025)
- [Preference optimization of protein language models as a multi-objective binder design paradigm](https://mlsanity.com/paper/2403.04187/) (2024)
- [EnerBridge-DPO: Energy-Guided Protein Inverse Folding with Markov Bridges and Direct Preference Optimization](https://mlsanity.com/paper/2506.09496/) (2025)

## Abstract (arXiv)

Direct Preference Optimization (DPO) is an effective approach for aligning protein language models with experimental design goals. However, DPO faces a scalability bottleneck: the number of possible training pairs grows quadratically with the number of labeled sequences, leading to prohibitive training times even for modestly sized datasets. We introduce g-DPO, a framework that (i) uses sequence space clustering to prune redundant pairs while preserving training signal, and (ii) amortizes likelihood computations with group-based approximations. Across three protein engineering tasks, g-DPO maintains in silico and in vitro performance that is statistically indistinguishable from standard DPO, while converging 1.7x to 5.4x times faster, with speedups that scale with dataset size and the structure of the underlying mutational landscape.