# Deciphering antibody affinity maturation with language models and weakly supervised learning

arXiv: https://arxiv.org/abs/2112.07782 · HTML: https://mlsanity.com/paper/2112.07782/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)

## Built on by (most cited first, 30 of 53)

- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders](https://mlsanity.com/paper/2203.12742/) (2022)
- [Conditional Antibody Design as 3D Equivariant Graph Translation](https://mlsanity.com/paper/2208.06073/) (2022)
- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein](https://mlsanity.com/paper/2401.06199/) (2024)
- [End-to-End Full-Atom Antibody Design](https://mlsanity.com/paper/2302.00203/) (2023)
- [AbDiffuser: Full-Atom Generation of in vitro Functioning Antibodies](https://mlsanity.com/paper/2308.05027/) (2023)
- [Antigen-Specific Antibody Design via Direct Energy-based Preference Optimization](https://mlsanity.com/paper/2403.16576/) (2024)
- [A Survey of Generative AI for de novo Drug Design: New Frontiers in Molecule and Protein Generation](https://mlsanity.com/paper/2402.08703/) (2024)
- [ProteinBench: A Holistic Evaluation of Protein Foundation Models](https://mlsanity.com/paper/2409.06744/) (2024)
- [Large scale paired antibody language models](https://mlsanity.com/paper/2403.17889/) (2024)
- [Data-Efficient Protein 3D Geometric Pretraining via Refinement of Diffused Protein Structure Decoy](https://mlsanity.com/paper/2302.10888/) (2023)
- [HELM: Hierarchical Encoding for mRNA Language Modeling](https://mlsanity.com/paper/2410.12459/) (2024)
- [dyAb: Flow Matching for Flexible Antibody Design with AlphaFold-driven Pre-binding Antigen](https://mlsanity.com/paper/2503.01910/) (2025)
- [Retrieval Augmented Diffusion Model for Structure-informed Antibody Design and Optimization](https://mlsanity.com/paper/2410.15040/) (2024)
- [Antibody Design and Optimization with Multi-scale Equivariant Graph Diffusion Models for Accurate Complex Antigen Binding](https://mlsanity.com/paper/2506.20957/) (2025)
- [Cross-Gate MLP with Protein Complex Invariant Embedding is A One-Shot Antibody Designer](https://mlsanity.com/paper/2305.09480/) (2023)
- [AsEP: Benchmarking Deep Learning Methods for Antibody-specific Epitope Prediction](https://mlsanity.com/paper/2407.18184/) (2024)
- [Linguistically inspired roadmap for building biologically reliable protein language models](https://mlsanity.com/paper/2207.00982/) (2022)
- [Computational Protein Science in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2501.10282/) (2025)
- [On Pre-trained Language Models for Antibody](https://mlsanity.com/paper/2301.12112/) (2023)
- [AbBiBench: A Benchmark for Antibody Binding Affinity Maturation and Design](https://mlsanity.com/paper/2506.04235/) (2025)
- [Incorporating Pre-training Paradigm for Antibody Sequence-Structure Co-design](https://mlsanity.com/paper/2211.08406/) (2022)
- [AVIDa-hIL6: A Large-Scale VHH Dataset Produced from an Immunized Alpaca for Predicting Antigen-Antibody Interactions](https://mlsanity.com/paper/2306.03329/) (2023)
- [A SARS-CoV-2 Interaction Dataset and VHH Sequence Corpus for Antibody Language Models](https://mlsanity.com/paper/2405.18749/) (2024)
- [Generative Humanization for Therapeutic Antibodies](https://mlsanity.com/paper/2412.04737/) (2024)
- [AbRank: A Benchmark Dataset and Metric-Learning Framework for Antibody-Antigen Affinity Ranking](https://mlsanity.com/paper/2506.17857/) (2025)
- [Speaking the Language of Science: Toward a General-Purpose Generative Foundation Model for the Natural Sciences](https://mlsanity.com/paper/2606.16905/) (2026)
- [ImmunoLingo: Linguistics-based formalization of the antibody language](https://mlsanity.com/paper/2209.12635/) (2022)
- [xTrimoABFold: De novo Antibody Structure Prediction without MSA](https://mlsanity.com/paper/2212.00735/) (2022)

## Abstract (arXiv)

In response to pathogens, the adaptive immune system generates specific antibodies that bind and neutralize foreign antigens. Understanding the composition of an individual's immune repertoire can provide insights into this process and reveal potential therapeutic antibodies. In this work, we explore the application of antibody-specific language models to aid understanding of immune repertoires. We introduce AntiBERTy, a language model trained on 558M natural antibody sequences. We find that within repertoires, our model clusters antibodies into trajectories resembling affinity maturation. Importantly, we show that models trained to predict highly redundant sequences under a multiple instance learning framework identify key binding residues in the process. With further development, the methods presented here will provide new insights into antigen binding from repertoire sequences alone.