mlsanityData to 14 Oct 2026

OPT: Open Pre-trained Transformer Language Models

arXiv 2205.01068 ↗First posted 2 May 2022cs.CL cs.LGcited by 3,512

Not on the map. Why?

Cited by 256 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.

Key facts. Cited by 3,512 papers in the map (data through 14 Oct 2026). Builds on 81 of its 120 references that are in the map. First posted on arXiv 2 May 2022.

Builds on

81 of 120 references.

References · 81

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
38 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2019Language Models as Knowledge Bases?cited by 1,936
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2017Decoupled Weight Decay Regularizationcited by 1,831
  13. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  14. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  15. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  16. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  17. 2018Datasheets for Datasetscited by 1,386
  18. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  19. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  20. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  21. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  22. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  23. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  24. 2017Mixed Precision Trainingcited by 1,156
  25. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  26. 2018Model Cards for Model Reportingcited by 1,080
  27. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  28. 2021Ethical and social risks of harm from Language Modelscited by 966
  29. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  30. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  31. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  32. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  33. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  34. 2020Recipes for building an open-domain chatbotcited by 593
  35. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  36. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  37. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  38. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 215; 2023: 1152; 2024: 1188; 2025: 637; 2026: 320.

3,512 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2023A Survey of Large Language Modelscited by 2,576
38 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  12. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  13. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  14. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  15. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  16. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  17. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  18. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  19. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  20. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  21. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  22. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  23. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  24. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  25. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  26. 2023BloombergGPT: A Large Language Model for Financecited by 939
  27. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  28. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  29. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  30. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  31. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  32. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  33. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  34. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  35. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  36. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  37. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  38. 2023StarCoder: may the source be with you!cited by 799

50 most cited of 3,512 All 3,512 (JSON)