mlsanityData to 7 Oct 2026

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

arXiv 2204.06745 ↗First posted 14 Apr 2022cs.CLcited by 711

Not on the map. Why?

Cited by 38 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe model's architecture and training and evaluate its performance on a range of language-understanding, mathematics, and knowledge-based tasks. We find that GPT-NeoX-20B is a particularly powerful few-shot reasoner and gains far more in performance when evaluated five-shot than similarly sized GPT-3 and FairSeq models. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.

Key facts. Cited by 711 papers in the map (data through 7 Oct 2026). Builds on 81 of its 142 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

81 of 142 references.

References · 81

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  12. 2017Decoupled Weight Decay Regularizationcited by 1,722
  13. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  14. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  15. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  16. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  17. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  18. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  19. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  20. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  21. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  22. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  23. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  24. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  25. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  26. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  27. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  28. 2019Green AIcited by 620
  29. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  30. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  31. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  32. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  33. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  34. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  35. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  36. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  37. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  38. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 70; 2023: 256; 2024: 210; 2025: 119; 2026: 56.

711 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  12. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
38 more
  1. 2022Mass-Editing Memory in a Transformercited by 976
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  4. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  6. 2023StarCoder: may the source be with you!cited by 790
  7. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  9. 2022Galactica: A Large Language Model for Sciencecited by 736
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  12. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  13. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  14. 2024A Survey on Large Language Models for Code Generationcited by 646
  15. 2023A Comprehensive Overview of Large Language Modelscited by 639
  16. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  17. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  18. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  19. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  20. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  21. 2023Textbooks Are All You Needcited by 561
  22. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  23. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  24. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  25. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  26. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  27. 2023Scaling Data-Constrained Language Modelscited by 391
  28. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  29. 2023Llemma: An Open Language Model For Mathematicscited by 385
  30. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  31. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  32. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  33. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  34. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  35. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  36. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  37. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  38. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 294

50 most cited of 711 All 711 (JSON)