# Incorporating Pre-training Paradigm for Antibody Sequence-Structure Co-design

arXiv: https://arxiv.org/abs/2211.08406 · HTML: https://mlsanity.com/paper/2211.08406/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 58 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design](https://mlsanity.com/paper/2110.04624/) (2021)
- [OntoProtein: Protein Pretraining With Gene Ontology Embedding](https://mlsanity.com/paper/2201.11147/) (2022)
- [Deciphering antibody affinity maturation with language models and weakly supervised learning](https://mlsanity.com/paper/2112.07782/) (2021)
- [Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design](https://mlsanity.com/paper/2106.13058/) (2021)
- [Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model](https://mlsanity.com/paper/2110.15527/) (2021)
- [Benchmarking deep generative models for diverse antibody sequence design](https://mlsanity.com/paper/2111.06801/) (2021)
- [ImmunoLingo: Linguistics-based formalization of the antibody language](https://mlsanity.com/paper/2209.12635/) (2022)

## Built on by (most cited first, 3 of 3)

- [AbDiffuser: Full-Atom Generation of in vitro Functioning Antibodies](https://mlsanity.com/paper/2308.05027/) (2023)
- [A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applications](https://mlsanity.com/paper/2403.00485/) (2024)
- [AI driven B-cell Immunotherapy Design](https://mlsanity.com/paper/2309.01122/) (2023)

## Abstract (arXiv)

Antibodies are versatile proteins that can bind to pathogens and provide effective protection for human body. Recently, deep learning-based computational antibody design has attracted popular attention since it automatically mines the antibody patterns from data that could be complementary to human experiences. However, the computational methods heavily rely on high-quality antibody structure data, which is quite limited. Besides, the complementarity-determining region (CDR), which is the key component of an antibody that determines the specificity and binding affinity, is highly variable and hard to predict. Therefore, the data limitation issue further raises the difficulty of CDR generation for antibodies. Fortunately, there exists a large amount of sequence data of antibodies that can help model the CDR and alleviate the reliance on structure data. By witnessing the success of pre-training models for protein modeling, in this paper, we develop the antibody pre-training language model and incorporate it into the (antigen-specific) antibody design model in a systemic way. Specifically, we first pre-train an antibody language model based on the sequence data, then propose a one-shot way for sequence and structure generation of CDR to avoid the heavy cost and error propagation from an autoregressive manner, and finally leverage the pre-trained antibody model for the antigen-specific antibody generation model with some carefully designed modules. Through various experiments, we show that our method achieves superior performances over previous baselines on different tasks, such as sequence and structure generation and antigen-binding CDR-H3 design.