# Interpreting and Steering Protein Language Models through Sparse Autoencoders

arXiv: https://arxiv.org/abs/2502.09135 · HTML: https://mlsanity.com/paper/2502.09135/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Interpreting biomedical and science models, one of 13 directions in Mechanistic interpretability.
- First posted on arXiv 13 Feb 2025.

## Placed in

- Mechanistic interpretability / [Interpreting biomedical and science models](https://mlsanity.com/mechinterp/interpreting-biomedical-and-science-models/)

## Built on by (most cited first, 17 of 17)

- [Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry](https://mlsanity.com/paper/2503.01822/) (2025)
- [AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features](https://mlsanity.com/paper/2510.00404/) (2025)
- [Sparse Autoencoders for Low-N Protein Function Prediction and Design](https://mlsanity.com/paper/2508.18567/) (2025)
- [Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse Autoencoders](https://mlsanity.com/paper/2508.09363/) (2025)
- [Protein Circuit Tracing via Cross-layer Transcoders](https://mlsanity.com/paper/2602.12026/) (2026)
- [Circuits, Features, and Heuristics in Molecular Transformers](https://mlsanity.com/paper/2512.09757/) (2025)
- [Two Stages of Folding: Convergent Mechanisms in AI Protein Folding Trunks](https://mlsanity.com/paper/2602.06020/) (2026)
- [Automated Neuron Labelling Enables Generative Steering and Interpretability in Protein Language Models](https://mlsanity.com/paper/2507.06458/) (2025)
- [GeoSAE: Geometric Prior-Guided Layer-Wise Sparse Autoencoder Annotation of Brain MRI Foundation Models](https://mlsanity.com/paper/2605.01829/) (2026)
- [Retrieval and competition: how a protein foundation model starts a protein](https://mlsanity.com/paper/2605.16331/) (2026)
- [Mechanistic Interpretability and Causal Feature Steering of Neural Quantum States via Sparse Autoencoders](https://mlsanity.com/paper/2607.01336/) (2026)
- [Quantifying the Role of OpenFold Components in Protein Structure Prediction](https://mlsanity.com/paper/2511.14781/) (2025)
- [Features Emerge as Discrete States: The First Application of SAEs to 3D Representations](https://mlsanity.com/paper/2512.11263/) (2025)
- [Circuit Tracing in Autoregressive Protein Language Models](https://mlsanity.com/paper/2606.16044/) (2026)
- [PairSAE: Mechanistic Interpretability from Pair Representations in Protein Co-Folding](https://mlsanity.com/paper/2606.27440/) (2026)
- [Interpreting Latent Protein Language Model Features with Geometric Annotations](https://mlsanity.com/paper/2608.26419/) (2026)
- [Generative modeling of intrinsically disordered protein regions by reinforcing sparse autoencoder features](https://mlsanity.com/paper/2610.02189/) (2026)

## Abstract (arXiv)

The rapid advancements in transformer-based language models have revolutionized natural language processing, yet understanding the internal mechanisms of these models remains a significant challenge. This paper explores the application of sparse autoencoders (SAE) to interpret the internal representations of protein language models, specifically focusing on the ESM-2 8M parameter model. By performing a statistical analysis on each latent component's relevance to distinct protein annotations, we identify potential interpretations linked to various protein characteristics, including transmembrane regions, binding sites, and specialized motifs. We then leverage these insights to guide sequence generation, shortlisting the relevant latent components that can steer the model towards desired targets such as zinc finger domains. This work contributes to the emerging field of mechanistic interpretability in biological sequence models, offering new perspectives on model steering for sequence design.