mlsanityData to 14 Oct 2026

PairSAE: Mechanistic Interpretability from Pair Representations in Protein Co-Folding

arXiv 2606.27440 ↗First posted 25 Jun 2026cs.LGcited by 0

On the map: Mechanistic interpretability → Interpreting biomedical and science models

Abstract (arXiv)

Foundation models for structural biology have achieved remarkable performance in predicting biomolecular structure and show promise for the design of proteins and small molecules. Yet understanding which internal features drive their outputs remains challenging. Standard sparse autoencoders (SAEs), effective on transformer-style sequence embeddings, do not transfer cleanly to pairformer-like architectures: naively operating on pairwise representations yields a quadratic blow-up of features and obscures concepts distributed jointly across sequence and pair representations. We introduce PairSAE, which summarizes pairwise tensors via an N-mode SVD into token-wise interaction roles, then uses a sparse autoencoder to learn a shared set of token-level features that decode into both sequence and pair representations. Evaluated on Boltz-2 activations for PLINDER protein-ligand complexes, PairSAE yields interpretable features that align with UniProt annotations and predict Boltz-2 affinity values. These results indicate that PairSAE links the latent space of foundation models for structural biology to interpretable structural concepts, clarifying what the model "knows" while avoiding pairformer-induced pitfalls that limit conventional SAEs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Interpreting biomedical and science models, one of 13 directions in Mechanistic interpretability. Builds on 11 of its 35 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

11 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Towards Interpretable Protein Structure Prediction with Sparse Autoencoders11 Mar
  2. 2025Interpreting and Steering Protein Language Models through Sparse Autoencoders13 Feb
  3. 2024InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders13 Nov

Other references · 8

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  2. 2022Toy Models of Superpositioncited by 971
  3. 2024Scaling and evaluating sparse autoencoderscited by 553
  4. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
  5. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 143
  6. 2024BatchTopK Sparse Autoencoderscited by 130
  7. 2022Polysemanticity and Capacity in Neural Networkscited by 77
  8. 2025Sparse Autoencoders for Hypothesis Generationcited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Systematic Evaluation of Co-folding Model Representations for Small-Molecule Learning
  2. 2025Triangle Multiplication Is All You Need For Biomolecular Structure Representations
  3. 2026Interpreting Latent Protein Language Model Features with Geometric Annotations
  4. 2024FoldToken2: Learning compact, invariant and generative protein structure language
  5. 2025ProteinAE: Protein Diffusion Autoencoders for Structure Encoding
  6. 2024Learning the Language of Protein Structure
  7. 2022Protein Sequence and Structure Co-Design with Equivariant Translation
  8. 2026From Snapshots to Symphonies: The Evolution of Protein Prediction from Static Structures to Generative Dynamics and Multimodal Interactions