mlsanityData to 7 Oct 2026

ProGen: Language Modeling for Protein Generation

arXiv 2004.03497 ↗First posted 8 Mar 2020q-bio.BM cs.LG stat.MLcited by 127

Not on the map. Why?

Abstract (arXiv)

Generative modeling for protein engineering is key to solving fundamental problems in synthetic biology, medicine, and material science. We pose protein engineering as an unsupervised sequence generation problem in order to leverage the exponentially growing set of proteins that lack costly, structural annotations. We train a 1.2B-parameter language model, ProGen, on ~280M protein sequences conditioned on taxonomic and keyword tags such as molecular function and cellular component. This provides ProGen with an unprecedented range of evolutionary sequence diversity and allows it to generate with fine-grained control as demonstrated by metrics based on primary sequence similarity, secondary structure accuracy, and conformational energy.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 21; 2023: 30; 2024: 22; 2025: 26; 2026: 10.

127 papers.

Sort:

By area:

  1. 2023Scaling Data-Constrained Language Modelscited by 391
  2. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  3. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  4. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  5. 2022ProGen2: Exploring the Boundaries of Protein Language Modelscited by 156
  6. 2022Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Modelscited by 97
  7. 2021Luna: Linear Unified Nested Attentioncited by 96
  8. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  9. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  10. 2023ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Textscited by 78
  11. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  12. 2024ProtAgents: Protein discovery via large language model multi-agent collaborations combining physics and machine learningcited by 63
38 more
  1. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  2. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  3. 2022Regression Transformer: Concurrent sequence regression and generation for molecular language modelingcited by 49
  4. 2021On the Expressive Power of Self-Attention Matricescited by 44
  5. 2023Ankh: Optimized Protein Language Model Unlocks General-Purpose Modellingcited by 44
  6. 2021Audio Transformerscited by 41
  7. 2022RITA: a Study on Scaling Up Generative Protein Sequence Modelscited by 39
  8. 2024AtomAgents: Alloy design and discovery through physics-aware multi-modal multi-agent artificial intelligencecited by 36
  9. 2021Protein sequence design with deep generative modelscited by 35
  10. 2024Full-Atom Peptide Design based on Multi-modal Flow Matchingcited by 34
  11. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  12. 2025Scientists' First Exam: Probing Cognitive Abilities of MLLM via Perception, Understanding, and Reasoningcited by 33
  13. 2023ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedbackcited by 30
  14. 2023A Latent Diffusion Model for Protein Structure Generationcited by 28
  15. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  16. 2021Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Designcited by 23
  17. 2023Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequencescited by 23
  18. 2021Question Generation for Adaptive Educationcited by 22
  19. 2023Symmetry-Informed Geometric Representation for Molecules, Proteins, and Crystalline Materialscited by 22
  20. 2021Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Modelcited by 20
  21. 2021A Generative Model for Raw Audio Using Transformer Architecturescited by 18
  22. 2021Deep Extrapolation for Attribute-Enhanced Generationcited by 18
  23. 2021Deep Generative Modeling for Protein Designcited by 17
  24. 2022Protein Language Models and Structure Prediction: Connection and Progressioncited by 17
  25. 2022Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMCcited by 17
  26. 2022Generative De Novo Protein Design with Global Contextcited by 15
  27. 2024Concept Bottleneck Language Models For protein designcited by 15
  28. 2021Cross-lingual Transfer of Monolingual Modelscited by 14
  29. 2021On Learning the Transformer Kernelcited by 14
  30. 2021Adaptive machine learning for protein engineeringcited by 13
  31. 2023A Hierarchical Training Paradigm for Antibody Structure-sequence Co-designcited by 13
  32. 2024Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?cited by 13
  33. 2023Importance Weighted Expectation-Maximization for Protein Sequence Designcited by 12
  34. 2024Training Compute-Optimal Protein Language Modelscited by 11
  35. 2022A Survey on Protein Representation Learning: Retrospect and Prospectcited by 10
  36. 2025BioMaze: Benchmarking and Enhancing Large Language Models for Biological Pathway Reasoningcited by 10
  37. 2022Improving Molecular Representation Learning with Metric Learning-enhanced Optimal Transportcited by 9
  38. 2022Generative power of a protein language model trained on multiple sequence alignmentscited by 9

50 most cited of 127 All 127 (JSON)