# Controlling Repetition in Protein Language Models

arXiv: https://arxiv.org/abs/2602.00782 · HTML: https://mlsanity.com/paper/2602.00782/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 31 Jan 2026.

## Builds on (most cited first, 30 of 32)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Learning to Generate Reviews and Discovering Sentiment](https://mlsanity.com/paper/1704.01444/) (2017)
- [Locally Typical Sampling](https://mlsanity.com/paper/2202.00666/) (2022)
- [Diffusion Language Models Are Versatile Protein Learners](https://mlsanity.com/paper/2402.18567/) (2024)
- [InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2412.12101/) (2024)
- [xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein](https://mlsanity.com/paper/2401.06199/) (2024)
- [Importance of Search and Evaluation Strategies in Neural Dialogue Modeling](https://mlsanity.com/paper/1811.00907/) (2018)
- [RITA: a Study on Scaling Up Generative Protein Sequence Models](https://mlsanity.com/paper/2205.05789/) (2022)
- [Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing](https://mlsanity.com/paper/2410.06331/) (2024)
- [Understanding the Repeat Curse in Large Language Models from a Feature Perspective](https://mlsanity.com/paper/2504.14218/) (2025)
- [Dissecting Fine-Tuning Unlearning in Large Language Models](https://mlsanity.com/paper/2410.06606/) (2024)
- [RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns](https://mlsanity.com/paper/2508.13152/) (2025)
- [Exploring the Personality Traits of LLMs through Latent Features Steering](https://mlsanity.com/paper/2410.10863/) (2024)
- [Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Models](https://mlsanity.com/paper/2508.10599/) (2025)
- [A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoning](https://mlsanity.com/paper/2406.12255/) (2024)
- [Concept Bottleneck Language Models For protein design](https://mlsanity.com/paper/2411.06090/) (2024)
- [Comparative Analysis of Pooling Mechanisms in LLMs: A Sentiment Analysis Perspective](https://mlsanity.com/paper/2411.14654/) (2024)
- [Towards Interpretable Protein Structure Prediction with Sparse Autoencoders](https://mlsanity.com/paper/2503.08764/) (2025)
- [MONICA: Real-Time Monitoring and Calibration of Chain-of-Thought Sycophancy in Large Reasoning Models](https://mlsanity.com/paper/2511.06419/) (2025)
- [PIXEL: Adaptive Steering Via Position-wise Injection with eXact Estimated Levels under Subspace Calibration](https://mlsanity.com/paper/2510.10205/) (2025)
- [Steering Protein Language Models](https://mlsanity.com/paper/2509.07983/) (2025)

## Built on by (most cited first, 5 of 5)

- [Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models](https://mlsanity.com/paper/2602.23179/) (2026)
- [Generative Modeling in Protein Design: Neural Representations, Conditional Generation, and Evaluation Standards](https://mlsanity.com/paper/2603.26378/) (2026)
- [Multi-Adapter Representation Interventions via Energy Calibration](https://mlsanity.com/paper/2605.28722/) (2026)
- [Unlocking Multimodal Protein Language Models at Inference Time](https://mlsanity.com/paper/2608.25855/) (2026)
- [Learning Latent Protein Languages for Autoregressive Generation](https://mlsanity.com/paper/2610.03978/) (2026)

## Abstract (arXiv)

Protein language models (PLMs) have enabled advances in structure prediction and de novo protein design, yet they frequently collapse into pathological repetition during generation. Unlike in text, where repetition merely reduces readability, in proteins it undermines structural confidence and functional viability. To unify this problem, we present the first systematic study of repetition in PLMs. We first propose quantitative metrics to characterize motif-level and homopolymer repetition and then demonstrate their negative impact on folding reliability. To address this challenge, we propose UCCS (Utility-Controlled Contrastive Steering), which steers protein generation with a constrained dataset. Instead of naively contrasting high- vs. low-repetition sequences, we construct contrastive sets that maximize differences in repetition while tightly controlling for structural utility. This disentanglement yields steering vectors that specifically target repetition without degrading foldability. Injected at inference, these vectors consistently reduce repetition without retraining or heuristic decoding. Experiments with ESM-3 and ProtGPT2 in CATH, UniRef50, and SCOP show that our method outperforms decoding penalties and other baselines, substantially lowering repetition while preserving AlphaFold confidence scores. Our results establish repetition control as a central challenge for PLMs and highlight dataset-guided steering as a principled approach for reliable protein generation.