# Large scale paired antibody language models

arXiv: https://arxiv.org/abs/2403.17889 · HTML: https://mlsanity.com/paper/2403.17889/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 54 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting](https://mlsanity.com/paper/2004.12651/) (2020)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [Deciphering antibody affinity maturation with language models and weakly supervised learning](https://mlsanity.com/paper/2112.07782/) (2021)
- [xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein](https://mlsanity.com/paper/2401.06199/) (2024)
- [Generative Antibody Design for Complementary Chain Pairing Sequences through Encoder-Decoder Language Model](https://mlsanity.com/paper/2301.02748/) (2023)

## Built on by (most cited first, 16 of 16)

- [HELM: Hierarchical Encoding for mRNA Language Modeling](https://mlsanity.com/paper/2410.12459/) (2024)
- [Multi-Modal CLIP-Informed Protein Editing](https://mlsanity.com/paper/2407.19296/) (2024)
- [Computational Protein Science in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2501.10282/) (2025)
- [Proteo-R1: Reasoning Foundation Models for De Novo Protein Design](https://mlsanity.com/paper/2605.02937/) (2026)
- [ABodyBuilder3: Improved and scalable antibody structure predictions](https://mlsanity.com/paper/2405.20863/) (2024)
- [A SARS-CoV-2 Interaction Dataset and VHH Sequence Corpus for Antibody Language Models](https://mlsanity.com/paper/2405.18749/) (2024)
- [VenusFactory: A Unified Platform for Protein Engineering Data Retrieval and Language Model Fine-Tuning](https://mlsanity.com/paper/2503.15438/) (2025)
- [Advances in Protein Representation Learning: Methods, Applications, and Future Directions](https://mlsanity.com/paper/2503.16659/) (2025)
- [S2ALM: Sequence-Structure Pre-trained Large Language Model for Comprehensive Antibody Representation Learning](https://mlsanity.com/paper/2411.15215/) (2024)
- [Leveraging Large Language Models to Predict Antibody Biological Activity Against Influenza A Hemagglutinin](https://mlsanity.com/paper/2502.00694/) (2025)
- [NbBench: Benchmarking Language Models for Comprehensive Nanobody Tasks](https://mlsanity.com/paper/2505.02022/) (2025)
- [Antibody Foundational Model : Ab-RoBERTa](https://mlsanity.com/paper/2506.13006/) (2025)
- [Sequence-based protein-protein interaction prediction and its applications in drug discovery](https://mlsanity.com/paper/2507.19805/) (2025)
- [Tokenizing Loops of Antibodies](https://mlsanity.com/paper/2509.08707/) (2025)
- [Conditionally Site-Independent Neural Evolution of Antibody Sequences](https://mlsanity.com/paper/2602.18982/) (2026)
- [Preference-based Antibody Expression Ranking: Scaling with Large-scale Weak Supervision](https://mlsanity.com/paper/2607.16263/) (2026)

## Abstract (arXiv)

Antibodies are proteins produced by the immune system that can identify and neutralise a wide variety of antigens with high specificity and affinity, and constitute the most successful class of biotherapeutics. With the advent of next-generation sequencing, billions of antibody sequences have been collected in recent years, though their application in the design of better therapeutics has been constrained by the sheer volume and complexity of the data. To address this challenge, we present IgBert and IgT5, the best performing antibody-specific language models developed to date which can consistently handle both paired and unpaired variable region sequences as input. These models are trained comprehensively using the more than two billion unpaired sequences and two million paired sequences of light and heavy chains present in the Observed Antibody Space dataset. We show that our models outperform existing antibody and protein language models on a diverse range of design and regression tasks relevant to antibody engineering. This advancement marks a significant leap forward in leveraging machine learning, large scale data sets and high-performance computing for enhancing antibody design for therapeutic development.