# What is a meaningful representation of protein sequences?

arXiv: https://arxiv.org/abs/2012.02679 · HTML: https://mlsanity.com/paper/2012.02679/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2020.

## Built on by (most cited first, 22 of 22)

- [The geometry of hidden representations of large transformer models](https://mlsanity.com/paper/2302.00294/) (2023)
- [Kermut: Composite kernel regression for protein variant effects](https://mlsanity.com/paper/2407.00002/) (2024)
- [Linguistically inspired roadmap for building biologically reliable protein language models](https://mlsanity.com/paper/2207.00982/) (2022)
- [Biological Sequence Kernels with Guaranteed Flexibility](https://mlsanity.com/paper/2304.03775/) (2023)
- [Knowledge-augmented Graph Machine Learning for Drug Discovery: A Survey](https://mlsanity.com/paper/2302.08261/) (2023)
- [Decoder ensembling for learned latent geometries](https://mlsanity.com/paper/2408.07507/) (2024)
- [Riemann2: Learning Riemannian Submanifolds from Riemannian Data](https://mlsanity.com/paper/2503.05540/) (2025)
- [Generative Enzyme Design Guided by Functionally Important Sites and Small-Molecule Substrates](https://mlsanity.com/paper/2405.08205/) (2024)
- [Identifying latent distances with Finslerian geometry](https://mlsanity.com/paper/2212.10010/) (2022)
- [Identifying Metric Structures of Deep Latent Variable Models](https://mlsanity.com/paper/2502.13757/) (2025)
- [ProteinAE: Protein Diffusion Autoencoders for Structure Encoding](https://mlsanity.com/paper/2510.10634/) (2025)
- [Interpreting Equivariant Representations](https://mlsanity.com/paper/2401.12588/) (2024)
- [Is an encoder within reach?](https://mlsanity.com/paper/2206.01552/) (2022)
- [On Probabilistic Pullback Metrics for Latent Hyperbolic Manifolds](https://mlsanity.com/paper/2410.20850/) (2024)
- [Long-context Protein Language Modeling Using Bidirectional Mamba with Shared Projection Layers](https://mlsanity.com/paper/2411.08909/) (2024)
- [PLM-eXplain: Divide and Conquer the Protein Embedding Space](https://mlsanity.com/paper/2504.07156/) (2025)
- [Training self-supervised peptide sequence models on artificially chopped proteins](https://mlsanity.com/paper/2211.06428/) (2022)
- [A Deep Learning Pipeline for Epilepsy Genomic Analysis Using GPT-2 XL and NVIDIA H100](https://mlsanity.com/paper/2510.00392/) (2025)
- [PepCompass: Navigating peptide embedding spaces using Riemannian Geometry](https://mlsanity.com/paper/2510.01988/) (2025)
- [Protein generation with embedding learning for motif diversification](https://mlsanity.com/paper/2510.18790/) (2025)
- [CryoProt: A Protein Pretraining Framework with Cross-Box Interactions on Cryo-EM Density Maps](https://mlsanity.com/paper/2606.00955/) (2026)
- [Protein contacts are already in the attention: a single-forward-pass alternative to the Categorical Jacobian](https://mlsanity.com/paper/2606.21876/) (2026)

## Abstract (arXiv)

How we choose to represent our data has a fundamental impact on our ability to subsequently extract information from them. Machine learning promises to automatically determine efficient representations from large unstructured datasets, such as those arising in biology. However, empirical evidence suggests that seemingly minor changes to these machine learning models yield drastically different data representations that result in different biological interpretations of data. This begs the question of what even constitutes the most meaningful representation. Here, we approach this question for representations of protein sequences, which have received considerable attention in the recent literature. We explore two key contexts in which representations naturally arise: transfer learning and interpretable learning. In the first context, we demonstrate that several contemporary practices yield suboptimal performance, and in the latter we demonstrate that taking representation geometry into account significantly improves interpretability and lets the models reveal biological information that is otherwise obscured.