mlsanityData to 14 Oct 2026

RITA: a Study on Scaling Up Generative Protein Sequence Models

arXiv 2205.05789 ↗First posted 11 May 2022q-bio.QM cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

In this work we introduce RITA: a suite of autoregressive generative models for protein sequences, with up to 1.2 billion parameters, trained on over 280 million protein sequences belonging to the UniRef-100 database. Such generative models hold the promise of greatly accelerating protein design. We conduct the first systematic study of how capabilities evolve with model size for autoregressive transformers in the protein domain: we evaluate RITA models in next amino acid prediction, zero-shot fitness, and enzyme function prediction, showing benefits from increased scale. We release the RITA models openly, to the benefit of the research community.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 13 of its 45 references that are in the map. First posted on arXiv 11 May 2022.

Builds on

13 of 45 references.

References · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2019Evaluating Protein Transfer Learning with TAPEcited by 231
  10. 2022Predictability and Surprise in Large Generative Modelscited by 210
  11. 2020ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computingcited by 162
  12. 2020ProGen: Language Modeling for Protein Generationcited by 128
1 more
  1. 2017Deep generative models of genetic variation capture mutation effectscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 9; 2024: 10; 2025: 9; 2026: 9.

42 papers.

Sort:

By area:

  1. 2022ProGen2: Exploring the Boundaries of Protein Language Modelscited by 159
  2. 2024Scientific Large Language Models: A Survey on Biological & Chemical Domainscited by 80
  3. 2022Tranception: protein fitness prediction with autoregressive transformers and inference-time retrievalcited by 79
  4. 2024xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Proteincited by 50
  5. 2023Ankh: Optimized Protein Language Model Unlocks General-Purpose Modellingcited by 45
  6. 2023PoET: A generative model of protein families as sequences-of-sequencescited by 29
  7. 2024A Hitchhiker's Guide to Scaling Law Estimationcited by 27
  8. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  9. 2025Protein Large Language Models: A Comprehensive Surveycited by 18
  10. 2022Protein Language Models and Structure Prediction: Connection and Progressioncited by 17
  11. 2022Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMCcited by 17
  12. 2023A Systematic Study of Joint Representation Learning on Protein Sequences and Structurescited by 15
30 more
  1. 2024Training Compute Thresholds: Features and Functions in AI Regulationcited by 14
  2. 2024Diffusion on language model encodings for protein sequence generationcited by 11
  3. 2024Generative AI for Controllable Protein Sequence Design: A Surveycited by 8
  4. 2025Computational Protein Science in the Era of Large Language Models (LLMs)cited by 6
  5. 2023Generative artificial intelligence for de novo protein designcited by 5
  6. 2025Protein Design with Dynamic Protein Vocabularycited by 5
  7. 2025PDFBench: A Benchmark for De novo Protein Design from Functioncited by 5
  8. 2026Controlling Repetition in Protein Language Modelscited by 5
  9. 2024AMPCliff: quantitative definition and benchmarking of activity cliffs in antimicrobial peptidescited by 3
  10. 2024Function-Guided Conditional Generation Using Protein Language Models with Adapterscited by 3
  11. 2025Large Language Models for Bioinformaticscited by 3
  12. 2023Is novelty predictable?cited by 2
  13. 2024Are Protein Language Models Compute Optimal?cited by 2
  14. 2025Prot2Token: A Unified Framework for Protein Modeling via Next-Token Predictioncited by 2
  15. 2022Materials Transformers Language Models for Generative Materials Design: a benchmark studycited by 1
  16. 2023ProtFIM: Fill-in-Middle Protein Sequence Design via Protein Language Modelscited by 1
  17. 2023Exploring the Protein Sequence Space with Global Generative Modelscited by 1
  18. 2023Multi-level Protein Representation Learning for Blind Mutational Effect Predictioncited by 1
  19. 2023Neural scaling laws for phenotypic drug discoverycited by 0
  20. 2025Leveraging Natural Language Processing to Unravel the Mystery of Life: A Review of NLP Approaches in Genomics, Transcriptomics, and Proteomicscited by 0
  21. 2025Directed Evolution of Proteins via Bayesian Optimization in Embedding Spacecited by 0
  22. 2025Discovery of Sustainable Refrigerants through Physics-Informed RL Fine-Tuning of Sequence Modelscited by 0
  23. 2026ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Designcited by 0
  24. 2026Two-Stage Fine-Tuning for Protein Sequence Generation with Targeted Amino-Acid Compositioncited by 0
  25. 2026Genotypic Triggers: Exposing Pharmacogenomic Blind Spots via Host-Specific Backdoors in Generative Antimicrobial Peptide Modelscited by 0
  26. 2026ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routingcited by 0
  27. 2026Transfer Calibrated Prediction Powered Inferencecited by 0
  28. 2026A General Harness for Protein Foundation Model Fitness Predictioncited by 0
  29. 2026Learning Latent Protein Languages for Autoregressive Generationcited by 0
  30. 2026Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Modelscited by 0