mlsanityData to 7 Oct 2026

Measuring Massive Multitask Language Understanding

arXiv 2009.03300 ↗First posted 7 Sep 2020cs.CY cs.AI cs.CLcited by 7,768

Not on the map. Why?

Cited by 1,030 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

Key facts. Cited by 7,768 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 49; 2023: 598; 2024: 2047; 2025: 2660; 2026: 2394.

7,768 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2024Qwen2 Technical Reportcited by 2,076
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2024Mixtral of Expertscited by 1,785
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  13. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  14. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  15. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  16. 2025Gemma 3 Technical Reportcited by 1,686
  17. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  18. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  19. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  20. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  21. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  22. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  23. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  24. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  25. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  26. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  27. 2024OpenAI o1 System Cardcited by 1,220
  28. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  29. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  30. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  31. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  32. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  33. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  34. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  35. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  36. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  37. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  38. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099

50 most cited of 7,768 All 7,768 (JSON)