# A SARS-CoV-2 Interaction Dataset and VHH Sequence Corpus for Antibody Language Models

arXiv: https://arxiv.org/abs/2405.18749 · HTML: https://mlsanity.com/paper/2405.18749/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 79 references that are in the map.
- First posted on arXiv 29 May 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [Deciphering antibody affinity maturation with language models and weakly supervised learning](https://mlsanity.com/paper/2112.07782/) (2021)
- [Reprogramming Pretrained Language Models for Antibody Sequence Infilling](https://mlsanity.com/paper/2210.07144/) (2022)
- [Large scale paired antibody language models](https://mlsanity.com/paper/2403.17889/) (2024)
- [AVIDa-hIL6: A Large-Scale VHH Dataset Produced from an Immunized Alpaca for Predicting Antigen-Antibody Interactions](https://mlsanity.com/paper/2306.03329/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP](https://mlsanity.com/paper/2505.16661/) (2025)
- [NbBench: Benchmarking Language Models for Comprehensive Nanobody Tasks](https://mlsanity.com/paper/2505.02022/) (2025)
- [Preference-based Antibody Expression Ranking: Scaling with Large-scale Weak Supervision](https://mlsanity.com/paper/2607.16263/) (2026)

## Abstract (arXiv)

Antibodies are crucial proteins produced by the immune system to eliminate harmful foreign substances and have become pivotal therapeutic agents for treating human diseases. To accelerate the discovery of antibody therapeutics, there is growing interest in constructing language models using antibody sequences. However, the applicability of pre-trained language models for antibody discovery has not been thoroughly evaluated due to the scarcity of labeled datasets. To overcome these limitations, we introduce AVIDa-SARS-CoV-2, a dataset featuring the antigen-variable domain of heavy chain of heavy chain antibody (VHH) interactions obtained from two alpacas immunized with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) spike proteins. AVIDa-SARS-CoV-2 includes binary labels indicating the binding or non-binding of diverse VHH sequences to 12 SARS-CoV-2 mutants, such as the Delta and Omicron variants. Furthermore, we release VHHCorpus-2M, a pre-training dataset for antibody language models, containing over two million VHH sequences. We report benchmark results for predicting SARS-CoV-2-VHH binding using VHHBERT pre-trained on VHHCorpus-2M and existing general protein and antibody-specific pre-trained language models. These results confirm that AVIDa-SARS-CoV-2 provides valuable benchmarks for evaluating the representation capabilities of antibody language models for binding prediction, thereby facilitating the development of AI-driven antibody discovery. The datasets are available at https://datasets.cognanous.com.