mlsanityData to 7 Oct 2026

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

arXiv 2304.01373 ↗First posted 3 Apr 2023cs.CLcited by 1,734

Not on the map. Why?

Cited by 128 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce textitPythia, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend textitPythia to facilitate research in many areas, and we present several case studies including novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics. Trained models, analysis code, training code, and training data can be found at urlhttps://github.com/EleutherAI/pythia.

Key facts. Cited by 1,734 papers in the map (data through 7 Oct 2026). Builds on 75 of its 101 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

75 of 101 references.

References · 75

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
38 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  9. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  13. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  14. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  15. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  16. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  17. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  18. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  19. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  20. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  21. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  22. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  23. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  24. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  25. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  26. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  27. 2018An Empirical Model of Large-Batch Trainingcited by 268
  28. 2021Scaling Laws for Transfercited by 265
  29. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  30. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  31. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  32. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  33. 2021The Values Encoded in Machine Learning Researchcited by 172
  34. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  35. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  36. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  37. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  38. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 231; 2024: 622; 2025: 446; 2026: 435.

1,734 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  11. 2023StarCoder: may the source be with you!cited by 790
  12. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
38 more
  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  3. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  4. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  5. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  7. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  8. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  9. 2023The Falcon Series of Open Language Modelscited by 526
  10. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  11. 2024Large Language Models: A Surveycited by 506
  12. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  13. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  14. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  15. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  16. 2023Scaling Data-Constrained Language Modelscited by 391
  17. 2023Llemma: An Open Language Model For Mathematicscited by 385
  18. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  19. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  20. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  21. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  22. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  23. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  24. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  25. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  26. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  27. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  28. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  29. 2023Language Models Represent Space and Timecited by 306
  30. 2023In-Context Learning Creates Task Vectorscited by 302
  31. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  32. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  33. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  34. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 283
  35. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  36. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  37. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  38. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256

50 most cited of 1,734 All 1,734 (JSON)