mlsanityData to 7 Oct 2026

OLMo: Accelerating the Science of Language Models

arXiv 2402.00838 ↗First posted 1 Feb 2024cs.CLcited by 599

Not on the map. Why?

Cited by 33 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, we have built OLMo, a competitive, truly Open Language Model, to enable the scientific study of language models. Unlike most prior efforts that have only released model weights and inference code, we release OLMo alongside open training data and training and evaluation code. We hope this release will empower the open research community and inspire a new wave of innovation.

Key facts. Cited by 599 papers in the map (data through 7 Oct 2026). Builds on 65 of its 99 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

65 of 99 references.

References · 65

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2016Layer Normalizationcited by 5,579
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
38 more
  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  13. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  15. 2024Mixtral of Expertscited by 1,785
  16. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  17. 2020GLU Variants Improve Transformercited by 1,724
  18. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  19. 2019Root Mean Square Layer Normalizationcited by 1,425
  20. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  21. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  22. 2017Mixed Precision Trainingcited by 1,135
  23. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  24. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  25. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  26. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  27. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  28. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  29. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  30. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  31. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  32. 2023The Falcon Series of Open Language Modelscited by 526
  33. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  34. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  35. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  36. 2023Scaling Data-Constrained Language Modelscited by 391
  37. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  38. 2016Using the Output Embedding to Improve Language Modelscited by 266

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 229; 2025: 228; 2026: 142.

599 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  4. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  5. 2024The Platonic Representation Hypothesiscited by 362
  6. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  7. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  8. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  9. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  10. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  11. 2024Generative Representational Instruction Tuningcited by 227
  12. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
38 more
  1. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  2. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  3. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  4. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  5. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  6. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  7. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 123
  8. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  9. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 114
  10. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  11. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 112
  12. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  13. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  14. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  15. 2025Agentic Large Language Models, a surveycited by 105
  16. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  17. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  18. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
  19. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  20. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  21. 2024Scaling Laws for Precisioncited by 92
  22. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  23. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 86
  24. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  25. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  26. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 82
  27. 2024Hyper-Connectionscited by 81
  28. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  29. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  30. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  31. 2025Self-Adapting Language Modelscited by 71
  32. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  33. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  34. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
  35. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  36. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  37. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  38. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66

50 most cited of 599 All 599 (JSON)