mlsanityData to 14 Oct 2026

ProGen2: Exploring the Boundaries of Protein Language Models

arXiv 2206.13517 ↗First posted 27 Jun 2022cs.LG q-bio.QMcited by 159

Not on the map. Why?

Abstract (arXiv)

Attention-based models trained on protein sequences have demonstrated incredible success at classification and generation tasks relevant for artificial intelligence-driven protein design. However, we lack a sufficient understanding of how very large-scale models and data play a role in effective protein model development. We introduce a suite of protein language models, named ProGen2, that are scaled up to 6.4B parameters and trained on different sequence datasets drawn from over a billion proteins from genomic, metagenomic, and immune repertoire databases. ProGen2 models show state-of-the-art performance in capturing the distribution of observed evolutionary sequences, generating novel viable sequences, and predicting protein fitness without additional finetuning. As large model sizes and raw numbers of protein sequences continue to become more widely accessible, our results suggest that a growing emphasis needs to be placed on the data distribution provided to a protein sequence model. We release the ProGen2 models and code at https://github.com/salesforce/progen.

Key facts. Cited by 159 papers in the map (data through 14 Oct 2026). Builds on 19 of its 73 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

19 of 73 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  12. 2020Learning from Protein Structure with Geometric Vector Perceptronscited by 313
7 more
  1. 2019Evaluating Protein Transfer Learning with TAPEcited by 231
  2. 2020ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computingcited by 162
  3. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  4. 2020ProGen: Language Modeling for Protein Generationcited by 128
  5. 2022Tranception: protein fitness prediction with autoregressive transformers and inference-time retrievalcited by 79
  6. 2021Deciphering antibody affinity maturation with language models and weakly supervised learningcited by 53
  7. 2022RITA: a Study on Scaling Up Generative Protein Sequence Modelscited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 23; 2024: 43; 2025: 41; 2026: 44.

159 papers.

Sort:

By area:

  1. 2023BloombergGPT: A Large Language Model for Financecited by 939
  2. 2022Galactica: A Large Language Model for Sciencecited by 744
  3. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
  4. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 98
  5. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  6. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  7. 2025Path Planning for Masked Diffusion Model Samplingcited by 80
  8. 2023ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Textscited by 78
  9. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  10. 2023Structure-informed Language Models Are Protein Designerscited by 68
  11. 2024InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoderscited by 66
  12. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 64
38 more
  1. 2024xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Proteincited by 50
  2. 2023Ankh: Optimized Protein Language Model Unlocks General-Purpose Modellingcited by 45
  3. 2023Graph Denoising Diffusion for Inverse Protein Foldingcited by 44
  4. 2024On the Scalability of GNNs for Molecular Graphscited by 36
  5. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  6. 2024Full-Atom Peptide Design based on Multi-modal Flow Matchingcited by 34
  7. 2024RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Taskscited by 33
  8. 2023PoET: A generative model of protein families as sequences-of-sequencescited by 29
  9. 2023A Latent Diffusion Model for Protein Structure Generationcited by 28
  10. 2024ProtT3: Protein-to-Text Generation for Text-based Protein Understandingcited by 25
  11. 2024ProteinBench: A Holistic Evaluation of Protein Foundation Modelscited by 25
  12. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  13. 2025GENERator: A Long-Context Generative Genomic Foundation Modelcited by 22
  14. 2022Reprogramming Pretrained Language Models for Antibody Sequence Infillingcited by 20
  15. 2022Protein Language Models and Structure Prediction: Connection and Progressioncited by 17
  16. 2022Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMCcited by 17
  17. 2025Planner Aware Path Learning in Diffusion Language Models Trainingcited by 14
  18. 2024Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequencescited by 13
  19. 2023Reprogramming Pretrained Language Models for Protein Sequence Representation Learningcited by 12
  20. 2024Training Compute-Optimal Protein Language Modelscited by 12
  21. 2025Gumbel-Softmax Flow Matching with Straight-Through Guidance for Controllable Biological Sequence Generationcited by 11
  22. 2025Elucidating the Design Space of Multimodal Protein Language Modelscited by 11
  23. 2025An All-Atom Generative Model for Designing Protein Complexescited by 10
  24. 2024Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competenciescited by 9
  25. 2024Hotspot-Driven Peptide Design via Multi-Fragment Autoregressive Extensioncited by 9
  26. 2024Generative AI for Controllable Protein Sequence Design: A Surveycited by 8
  27. 2025CFP-Gen: Combinatorial Functional Protein Generation via Diffusion Language Modelscited by 8
  28. 2022Linguistically inspired roadmap for building biologically reliable protein language modelscited by 7
  29. 2023Retrieved Sequence Augmentation for Protein Representation Learningcited by 7
  30. 2023Fast and Functional Structured Data Generators Rooted in Out-of-Equilibrium Physicscited by 7
  31. 2023Materials Informatics Transformer: A Language Model for Interpretable Materials Properties Predictioncited by 7
  32. 2024ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modelingcited by 7
  33. 2024MSAGPT: Neural Prompting Protein Structure Prediction via MSA Generative Pre-Trainingcited by 7
  34. 2024SciDFM: A Large Language Model with Mixture-of-Experts for Sciencecited by 7
  35. 2025Towards A Universal Graph Structural Encodercited by 7
  36. 2025Steering Generative Models with Experimental Data for Protein Fitness Optimizationcited by 7
  37. 2025Steering Protein Language Modelscited by 7
  38. 2025A Diffusion Model to Shrink Proteins While Maintaining Their Functioncited by 7

50 most cited of 159 All 159 (JSON)