mlsanityData to 7 Oct 2026

OLMES: A Standard for Language Model Evaluations

arXiv 2406.08446 ↗First posted 12 Jun 2024cs.CL cs.AIcited by 100

Not on the map. Why?

Abstract (arXiv)

Progress in AI is often demonstrated by new models claiming improved performance on tasks measuring model capabilities. Evaluating language models can be particularly challenging, as choices of how a model is evaluated on a task can lead to large changes in measured performance. There is no common standard setup, so different models are evaluated on the same tasks in different ways, leading to claims about which models perform best not being reproducible. We propose OLMES, a completely documented, practical, open standard for reproducible LLM evaluations. In developing this standard, we identify and review the varying factors in evaluation practices adopted by the community - such as details of prompt formatting, choice of in-context examples, probability normalizations, and task formulation. In particular, OLMES supports meaningful comparisons between smaller base models that require the unnatural "cloze" formulation of multiple-choice questions against larger models that can utilize the original formulation. OLMES includes well-considered, documented recommendations guided by results from existing literature as well as new experiments resolving open questions.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

28 of 50 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
16 more
  1. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  2. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  3. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  4. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  6. 2023The Falcon Series of Open Language Modelscited by 526
  7. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  8. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  9. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  10. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  11. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  12. 2024Stable LM 2 1.6B Technical Reportcited by 86
  13. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 66
  14. 2024A Study on Large Language Models' Limitations in Multiple-Choice Question Answeringcited by 22
  15. 2024Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AIcited by 18
  16. 2023Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracycited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 40; 2026: 53.

100 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  2. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  3. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  4. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 33
  5. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  6. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  7. 2025Fluid Language Model Benchmarkingcited by 30
  8. 2025Scaling Laws Are Unreliable for Downstream Tasks: A Reality Checkcited by 27
  9. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 25
  10. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
  11. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
  12. 2025DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluationcited by 21
38 more
  1. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 21
  2. 2025Metadata Conditioning Accelerates Language Model Pre-trainingcited by 20
  3. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingcited by 19
  4. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  5. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  6. 2025Train-before-Test Harmonizes Language Model Rankingscited by 12
  7. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 12
  8. 2026Adaptive Loops and Memory in Transformers: Think Harder or Know More?cited by 11
  9. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  10. 2026Sparse Layers are Critical to Scaling Looped Language Modelscited by 10
  11. 2026Prescriptive Scaling Laws for Data Constrained Trainingcited by 9
  12. 2025PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflictscited by 8
  13. 2025Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarkscited by 8
  14. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  15. 2025SkyLadder: Better and Faster Pretraining via Context Window Schedulingcited by 6
  16. 2025Efficient Model Development through Fine-tuning Transfercited by 6
  17. 2025Train Once, Answer All: Many Pretraining Experiments for the Cost of Onecited by 6
  18. 2025Relative Scaling Laws for LLMscited by 5
  19. 2024Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?cited by 4
  20. 2025Curriculum-Guided Layer Scaling for Language Model Pretrainingcited by 4
  21. 2025Low-rank Momentum Factorization for Memory Efficient Trainingcited by 4
  22. 2025Pre-Training LLMs on a budget: A comparison of three optimizerscited by 4
  23. 2025Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoningcited by 4
  24. 2025Typhoon T1: An Open Thai Reasoning Modelcited by 3
  25. 2025FilBench: Can LLMs Understand and Generate Filipino?cited by 3
  26. 2026Proxy Compression for Language Modelingcited by 3
  27. 2026"UberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Datasetcited by 3
  28. 2026AdaSplash-2: Faster Differentiable Sparse Attentioncited by 3
  29. 2026DashAttention: Differentiable and Adaptive Sparse Hierarchical Attentioncited by 3
  30. 2026LoopMTP: A looped transformer guided by latent multi-token predictioncited by 3
  31. 2024Neutral Residues: Revisiting Adapters for Model Extensioncited by 2
  32. 2025Choices Speak Louder than Questionscited by 2
  33. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  34. 2026Native Reasoning Models: Training Language Models to Reason on Unverifiable Datacited by 2
  35. 2026A Dual-Path Architecture for Scaling Compute and Capacity in LLMscited by 2
  36. 2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
  37. 2024PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Modelscited by 1
  38. 2025Optimal Formats for Weight Quantisationcited by 1

50 most cited of 100 All 100 (JSON)