Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models
Not on the map. Why?
Abstract (arXiv)
Protein sequences are abundant in repeating segments, both as exact copies and as approximate segments with mutations. These repeats are important for protein structure and function, motivating decades of algorithmic work on repeat identification. Recent work has shown that protein language models (PLMs) identify repeats, by examining their behavior in masked-token prediction. To elucidate their internal mechanisms, we investigate how PLMs detect both exact and approximate repeats. We find that the mechanism for approximate repeats functionally subsumes that of exact repeats. We then characterize this mechanism, revealing two main stages: PLMs first build feature representations using both general positional attention heads and biologically specialized components, such as neurons that encode amino-acid similarity. Then, induction heads attend to aligned tokens across repeated segments, promoting the correct answer. Our results reveal how PLMs solve this biological task by combining language-based pattern matching with specialized biological knowledge, thereby establishing a basis for studying more complex evolutionary processes in PLMs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 67 references that are in the map. First posted on arXiv 26 Feb 2026.
Builds on
28 of 67 references.
References · 28
- 2020GLU Variants Improve Transformercited by 1,724
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
- 2022In-context Learning and Induction Headscited by 930
- 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
- 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
- 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
- 2025Why do LLMs attend to the first token?cited by 122
- 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
- 2020BERTology Meets Biology: Interpreting Attention in Protein Language Modelscited by 85
- 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
16 more
- 2024InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoderscited by 64
- 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
- 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 46
- 2024Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learningcited by 46
- 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
- 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 22
- 2025The Dual-Route Model of Inductioncited by 22
- 2022Protein Language Models and Structure Prediction: Connection and Progressioncited by 17
- 2024Concept Bottleneck Language Models For protein designcited by 15
- 2025Mechanistic Interpretability as Statistical Estimation: A Variance Analysiscited by 14
- 2025Towards Interpretable Protein Structure Prediction with Sparse Autoencoderscited by 12
- 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11
- 2025Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learningcited by 5
- 2026Controlling Repetition in Protein Language Modelscited by 4
- 2025ProtSAE: Disentangling and Interpreting Protein Language Models via Semantically-Guided Sparse Autoencoderscited by 3
- 2025Reverse-Engineering the Retrieval Process in GenIR Modelscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.