# Towards Interpretable Protein Structure Prediction with Sparse Autoencoders

arXiv: https://arxiv.org/abs/2503.08764 · HTML: https://mlsanity.com/paper/2503.08764/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Interpreting biomedical and science models, one of 13 directions in Mechanistic interpretability.
- Builds on 7 of its 28 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Placed in

- Mechanistic interpretability / [Interpreting biomedical and science models](https://mlsanity.com/mechinterp/interpreting-biomedical-and-science-models/)

## Builds on (most cited first, 7 of 7)

- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Autoencoders Trained on the Same Data Learn Different Features](https://mlsanity.com/paper/2501.16615/) (2025)
- [BERTology Meets Biology: Interpreting Attention in Protein Language Models](https://mlsanity.com/paper/2006.15222/) (2020)
- [Learning Ordered Representations with Nested Dropout](https://mlsanity.com/paper/1402.0915/) (2014)
- [InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2412.12101/) (2024)
- [Anytime Sampling for Autoregressive Models via Ordered Autoencoding](https://mlsanity.com/paper/2102.11495/) (2021)

## Built on by (most cited first, 12 of 12)

- [Controlling Repetition in Protein Language Models](https://mlsanity.com/paper/2602.00782/) (2026)
- [Sparse Autoencoders for Low-N Protein Function Prediction and Design](https://mlsanity.com/paper/2508.18567/) (2025)
- [Protein Circuit Tracing via Cross-layer Transcoders](https://mlsanity.com/paper/2602.12026/) (2026)
- [Unveiling Latent Knowledge in Chemistry Language Models through Sparse Autoencoders](https://mlsanity.com/paper/2512.08077/) (2025)
- [Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models](https://mlsanity.com/paper/2602.23179/) (2026)
- [Retrieval and competition: how a protein foundation model starts a protein](https://mlsanity.com/paper/2605.16331/) (2026)
- [VFUSE: Virulent Feature Understanding with Sparse autoEncoders](https://mlsanity.com/paper/2606.10080/) (2026)
- [Mechanistic Interpretability of Antibody Language Models Using SAEs](https://mlsanity.com/paper/2512.05794/) (2025)
- [From Snapshots to Symphonies: The Evolution of Protein Prediction from Static Structures to Generative Dynamics and Multimodal Interactions](https://mlsanity.com/paper/2603.18505/) (2026)
- [Circuit Tracing in Autoregressive Protein Language Models](https://mlsanity.com/paper/2606.16044/) (2026)
- [PairSAE: Mechanistic Interpretability from Pair Representations in Protein Co-Folding](https://mlsanity.com/paper/2606.27440/) (2026)
- [Interpreting Latent Protein Language Model Features with Geometric Annotations](https://mlsanity.com/paper/2608.26419/) (2026)

## Abstract (arXiv)

Protein language models have revolutionized structure prediction, but their nonlinear nature obscures how sequence representations inform structure prediction. While sparse autoencoders (SAEs) offer a path to interpretability here by learning linear representations in high-dimensional space, their application has been limited to smaller protein language models unable to perform structure prediction. In this work, we make two key advances: (1) we scale SAEs to ESM2-3B, the base model for ESMFold, enabling mechanistic interpretability of protein structure prediction for the first time, and (2) we adapt Matryoshka SAEs for protein language models, which learn hierarchically organized features by forcing nested groups of latents to reconstruct inputs independently. We demonstrate that our Matryoshka SAEs achieve comparable or better performance than standard architectures. Through comprehensive evaluations, we show that SAEs trained on ESM2-3B significantly outperform those trained on smaller models for both biological concept discovery and contact map prediction. Finally, we present an initial case study demonstrating how our approach enables targeted steering of ESMFold predictions, increasing structure solvent accessibility while fixing the input sequence. To facilitate further investigation by the broader community, we open-source our code, dataset, pretrained models https://github.com/johnyang101/reticular-sae , and visualizer https://sae.reticular.ai .