# Reprogramming Pretrained Language Models for Antibody Sequence Infilling

arXiv: https://arxiv.org/abs/2210.07144 · HTML: https://mlsanity.com/paper/2210.07144/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 73 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2205.15019/) (2022)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design](https://mlsanity.com/paper/2110.04624/) (2021)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)
- [Conditional Antibody Design as 3D Equivariant Graph Translation](https://mlsanity.com/paper/2208.06073/) (2022)
- [Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning](https://mlsanity.com/paper/2202.10629/) (2022)
- [Cross-modal Adversarial Reprogramming](https://mlsanity.com/paper/2102.07325/) (2021)
- [When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute](https://mlsanity.com/paper/2102.12459/) (2021)
- [Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design](https://mlsanity.com/paper/2106.13058/) (2021)
- [Improving Molecular Design by Stochastic Iterative Target Augmentation](https://mlsanity.com/paper/2002.04720/) (2020)
- [AntBO: Towards Real-World Automated Antibody Design with Combinatorial Bayesian Optimisation](https://mlsanity.com/paper/2201.12570/) (2022)
- [Reprogramming Language Models for Molecular Representation Learning](https://mlsanity.com/paper/2012.03460/) (2020)
- [Benchmarking deep generative models for diverse antibody sequence design](https://mlsanity.com/paper/2111.06801/) (2021)

## Built on by (most cited first, 20 of 20)

- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Diffusion Language Models Are Versatile Protein Learners](https://mlsanity.com/paper/2402.18567/) (2024)
- [LLMs are Good Sign Language Translators](https://mlsanity.com/paper/2404.00925/) (2024)
- [Reprogramming Pretrained Target-Specific Diffusion Models for Dual-Target Drug Design](https://mlsanity.com/paper/2410.20688/) (2024)
- [Adversarial Machine Learning for Social Good: Reframing the Adversary as an Ally](https://mlsanity.com/paper/2310.03614/) (2023)
- [Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classification](https://mlsanity.com/paper/2412.16780/) (2024)
- [Computational Protein Science in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2501.10282/) (2025)
- [Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction](https://mlsanity.com/paper/2506.04650/) (2025)
- [A SARS-CoV-2 Interaction Dataset and VHH Sequence Corpus for Antibody Language Models](https://mlsanity.com/paper/2405.18749/) (2024)
- [Antibody DomainBed: Out-of-Distribution Generalization in Therapeutic Protein Design](https://mlsanity.com/paper/2407.21028/) (2024)
- [SeisMoLLM: Advancing Seismic Monitoring via Cross-modal Transfer with Pre-trained Large Language Model](https://mlsanity.com/paper/2502.19960/) (2025)
- [SFM-Protein: Integrative Co-evolutionary Pre-training for Advanced Protein Sequence Representation](https://mlsanity.com/paper/2410.24022/) (2024)
- [S2ALM: Sequence-Structure Pre-trained Large Language Model for Comprehensive Antibody Representation Learning](https://mlsanity.com/paper/2411.15215/) (2024)
- [Model Reprogramming Demystified: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2506.00620/) (2025)
- [Prompting the Unseen: Detecting Hidden Backdoors in Black-Box Models](https://mlsanity.com/paper/2411.09540/) (2024)
- [Accelerating Automatic Program Repair with Dual Retrieval-Augmented Fine-Tuning and Patch Generation on Large Language Models](https://mlsanity.com/paper/2507.10103/) (2025)
- [EnzyControl: Adding Functional and Substrate-Specific Control for Enzyme Backbone Generation](https://mlsanity.com/paper/2510.25132/) (2025)
- [Decoupled Sequence and Structure Generation for Realistic Antibody Design](https://mlsanity.com/paper/2402.05982/) (2024)
- [Generalizing to Out-of-Sample Degradations via Model Reprogramming](https://mlsanity.com/paper/2403.05886/) (2024)
- [BOAT: Navigating the Sea of In Silico Predictors for Antibody Design via Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2604.13980/) (2026)

## Abstract (arXiv)

Antibodies comprise the most versatile class of binding molecules, with numerous applications in biomedicine. Computational design of antibodies involves generating novel and diverse sequences, while maintaining structural consistency. Unique to antibodies, designing the complementarity-determining region (CDR), which determines the antigen binding affinity and specificity, creates its own unique challenges. Recent deep learning models have shown impressive results, however the limited number of known antibody sequence/structure pairs frequently leads to degraded performance, particularly lacking diversity in the generated sequences. In our work we address this challenge by leveraging Model Reprogramming (MR), which repurposes pretrained models on a source language to adapt to the tasks that are in a different language and have scarce data - where it may be difficult to train a high-performing model from scratch or effectively fine-tune an existing pre-trained model on the specific task. Specifically, we introduce ReprogBert in which a pretrained English language model is repurposed for protein sequence infilling - thus considers cross-language adaptation using less data. Results on antibody design benchmarks show that our model on low-resourced antibody sequence dataset provides highly diverse CDR sequences, up to more than a two-fold increase of diversity over the baselines, without losing structural integrity and naturalness. The generated sequences also demonstrate enhanced antigen binding specificity and virus neutralization ability. Code is available at https://github.com/IBM/ReprogBERT