# S2ALM: Sequence-Structure Pre-trained Large Language Model for Comprehensive Antibody Representation Learning

arXiv: https://arxiv.org/abs/2411.15215 · HTML: https://mlsanity.com/paper/2411.15215/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 69 references that are in the map.
- First posted on arXiv 20 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [ProGen2: Exploring the Boundaries of Protein Language Models](https://mlsanity.com/paper/2206.13517/) (2022)
- [Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design](https://mlsanity.com/paper/2110.04624/) (2021)
- [Deciphering antibody affinity maturation with language models and weakly supervised learning](https://mlsanity.com/paper/2112.07782/) (2021)
- [xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein](https://mlsanity.com/paper/2401.06199/) (2024)
- [Structure-aware Protein Self-supervised Learning](https://mlsanity.com/paper/2204.04213/) (2022)
- [Reprogramming Pretrained Language Models for Antibody Sequence Infilling](https://mlsanity.com/paper/2210.07144/) (2022)
- [Large scale paired antibody language models](https://mlsanity.com/paper/2403.17889/) (2024)
- [Bridge-IF: Learning Inverse Protein Folding with Markov Bridges](https://mlsanity.com/paper/2411.02120/) (2024)
- [Multi-Modal CLIP-Informed Protein Editing](https://mlsanity.com/paper/2407.19296/) (2024)
- [Antibody Representation Learning for Drug Discovery](https://mlsanity.com/paper/2210.02881/) (2022)
- [On Pre-trained Language Models for Antibody](https://mlsanity.com/paper/2301.12112/) (2023)
- [ECRECer: Enzyme Commission Number Recommendation and Benchmarking based on Multiagent Dual-core Learning](https://mlsanity.com/paper/2202.03632/) (2022)

## Built on by (most cited first, 2 of 2)

- [Conformation-Aware Structure Prediction of Antigen-Recognizing Immune Proteins](https://mlsanity.com/paper/2507.09054/) (2025)
- [Antigen-specific Antibody Multi-modal Foundation Model for Functional Antibody Design](https://mlsanity.com/paper/2607.20057/) (2026)

## Abstract (arXiv)

Antibodies safeguard our health through their precise and potent binding to specific antigens, demonstrating promising therapeutic efficacy in the treatment of numerous diseases, including COVID-19. Recent advancements in biomedical language models have shown the great potential to interpret complex biological structures and functions. However, existing antibody specific models have a notable limitation that they lack explicit consideration for antibody structural information, despite the fact that both 1D sequence and 3D structure carry unique and complementary insights into antibody behavior and functionality. This paper proposes Sequence-Structure multi-level pre-trained Antibody Language Model (S2ALM), combining holistic sequential and structural information in one unified, generic antibody foundation model. We construct a hierarchical pre-training paradigm incorporated with two customized multi-level training objectives to facilitate the modeling of comprehensive antibody representations. S2ALM's representation space uncovers inherent functional binding mechanisms, biological evolution properties and structural interaction patterns. Pre-trained over 75 million sequences and 11.7 million structures, S2ALM can be adopted for diverse downstream tasks: accurately predicting antigen-antibody binding affinities, precisely distinguishing B cell maturation stages, identifying antibody crucial binding positions, and specifically designing novel coronavirus-binding antibodies. Remarkably, S2ALM outperforms well-established and renowned baselines and sets new state-of-the-art performance across extensive antibody specific understanding and generation tasks. S2ALM's ability to model comprehensive and generalized representations further positions its potential to advance real-world therapeutic antibody development, potentially addressing unmet academic, industrial, and clinical needs.