# Ankh: Optimized Protein Language Model Unlocks General-Purpose Modelling

arXiv: https://arxiv.org/abs/2301.06568 · HTML: https://mlsanity.com/paper/2301.06568/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 60 references that are in the map.
- First posted on arXiv 16 Jan 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Evaluating Protein Transfer Learning with TAPE](https://mlsanity.com/paper/1906.08230/) (2019)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Rethinking Positional Encoding in Language Pre-training](https://mlsanity.com/paper/2006.15595/) (2020)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [ProGen: Language Modeling for Protein Generation](https://mlsanity.com/paper/2004.03497/) (2020)
- [ConvBERT: Improving BERT with Span-based Dynamic Convolution](https://mlsanity.com/paper/2008.02496/) (2020)
- [BERTology Meets Biology: Interpreting Attention in Protein Language Models](https://mlsanity.com/paper/2006.15222/) (2020)
- [PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding](https://mlsanity.com/paper/2206.02096/) (2022)
- [PMI-Masking: Principled masking of correlated spans](https://mlsanity.com/paper/2010.01825/) (2020)
- [DeepSF: deep convolutional neural network for mapping protein sequences to folds](https://mlsanity.com/paper/1706.01010/) (2017)
- [RITA: a Study on Scaling Up Generative Protein Sequence Models](https://mlsanity.com/paper/2205.05789/) (2022)
- [ProteinNet: a standardized data set for machine learning of protein structure](https://mlsanity.com/paper/1902.00249/) (2019)
- [Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks](https://mlsanity.com/paper/2010.09458/) (2020)
- [Analysing the Effect of Masking Length Distribution of MLM: An Evaluation Framework and Case Study on Chinese MRC Datasets](https://mlsanity.com/paper/2110.15712/) (2021)

## Built on by (most cited first, 30 of 45)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein](https://mlsanity.com/paper/2401.06199/) (2024)
- [Protein Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.17504/) (2025)
- [A Systematic Study of Joint Representation Learning on Protein Sequences and Structures](https://mlsanity.com/paper/2303.06275/) (2023)
- [Training Compute-Optimal Protein Language Models](https://mlsanity.com/paper/2411.02142/) (2024)
- [Endowing Protein Language Models with Structural Knowledge](https://mlsanity.com/paper/2401.14819/) (2024)
- [Structure-Informed Protein Language Model](https://mlsanity.com/paper/2402.05856/) (2024)
- [Simple, Efficient and Scalable Structure-aware Adapter Boosts Protein Language Models](https://mlsanity.com/paper/2404.14850/) (2024)
- [Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods](https://mlsanity.com/paper/2411.17669/) (2024)
- [Computational Protein Science in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2501.10282/) (2025)
- [VenusX: Unlocking Fine-Grained Functional Understanding of Proteins](https://mlsanity.com/paper/2505.11812/) (2025)
- [Protein Representation Learning with Sequence Information Embedding: Does it Always Lead to a Better Performance?](https://mlsanity.com/paper/2406.19755/) (2024)
- [Immunogenicity Prediction with Dual Attention Enables Vaccine Target Selection](https://mlsanity.com/paper/2410.02647/) (2024)
- [PFMBench: Protein Foundation Model Benchmark](https://mlsanity.com/paper/2506.14796/) (2025)
- [Ophiuchus: Scalable Modeling of Protein Structures through Hierarchical Coarse-graining SO(3)-Equivariant Autoencoders](https://mlsanity.com/paper/2310.02508/) (2023)
- [VenusFactory: A Unified Platform for Protein Engineering Data Retrieval and Language Model Fine-Tuning](https://mlsanity.com/paper/2503.15438/) (2025)
- [Ankh3: Multi-Task Pretraining with Sequence Denoising and Completion Enhances Protein Representations](https://mlsanity.com/paper/2505.20052/) (2025)
- [Sequence-Only Prediction of Binding Affinity Changes: A Robust and Interpretable Model for Antibody Engineering](https://mlsanity.com/paper/2505.20301/) (2025)
- [Are Protein Language Models Compute Optimal?](https://mlsanity.com/paper/2406.07249/) (2024)
- [SeqProFT: Sequence-only Protein Property Prediction with LoRA Finetuning](https://mlsanity.com/paper/2411.11530/) (2024)
- [Beyond Simple Concatenation: Fairly Assessing PLM Architectures for Multi-Chain Protein-Protein Interactions Prediction](https://mlsanity.com/paper/2505.20036/) (2025)
- [Prot2Token: A Unified Framework for Protein Modeling via Next-Token Prediction](https://mlsanity.com/paper/2505.20589/) (2025)
- [Topotein: Topological Deep Learning for Protein Representation Learning](https://mlsanity.com/paper/2509.03885/) (2025)
- [Towards Understanding the Shape of Representations in Protein Language Models](https://mlsanity.com/paper/2509.24895/) (2025)
- [Multimodal Protein Language Models for Enzyme Kinetic Parameters: From Substrate Recognition to Conformational Adaptation](https://mlsanity.com/paper/2603.12845/) (2026)
- [Exploring the Protein Sequence Space with Global Generative Models](https://mlsanity.com/paper/2305.01941/) (2023)
- [Contrastive learning of T cell receptor representations](https://mlsanity.com/paper/2406.06397/) (2024)
- [Fast and Interpretable Protein Substructure Alignment via Optimal Transport](https://mlsanity.com/paper/2510.11752/) (2025)
- [Generalization Beyond Benchmarks: Evaluating Learnable Protein-Ligand Scoring Functions on Unseen Targets](https://mlsanity.com/paper/2512.05386/) (2025)

## Abstract (arXiv)

As opposed to scaling-up protein language models (PLMs), we seek improving performance via protein-specific optimization. Although the proportionality between the language model size and the richness of its learned representations is validated, we prioritize accessibility and pursue a path of data-efficient, cost-reduced, and knowledge-guided optimization. Through over twenty experiments ranging from masking, architecture, and pre-training data, we derive insights from protein-specific experimentation into building a model that interprets the language of life, optimally. We present Ankh, the first general-purpose PLM trained on Google's TPU-v4 surpassing the state-of-the-art performance with fewer parameters (<10% for pre-training, <7% for inference, and <30% for the embedding dimension). We provide a representative range of structure and function benchmarks where Ankh excels. We further provide a protein variant generation analysis on High-N and One-N input data scales where Ankh succeeds in learning protein evolutionary conservation-mutation trends and introducing functional diversity while retaining key structural-functional characteristics. We dedicate our work to promoting accessibility to research innovation via attainable resources.