mlsanityData to 14 Oct 2026

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

arXiv 2206.04615 ↗First posted 9 Jun 2022cs.CL cs.AI cs.CYcited by 1,998

Not on the map. Why?

Cited by 419 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

Key facts. Cited by 1,998 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Jun 2022.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 106; 2023: 499; 2024: 598; 2025: 447; 2026: 348.

1,998 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,856
38 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2025s1: Simple test-time scalingcited by 1,341
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  10. 2023PaLM 2 Technical Reportcited by 1,162
  11. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  13. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  14. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  15. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  16. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  17. 2023BloombergGPT: A Large Language Model for Financecited by 939
  18. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  19. 2023Large Language Models as Optimizerscited by 864
  20. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  21. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  22. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  23. 2023Large Language Modelscited by 780
  24. 2023Whose Opinions Do Language Models Reflect?cited by 773
  25. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  26. 2022A Survey on In-context Learningcited by 721
  27. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  28. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  29. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  30. 2022Large Language Models Can Self-Improvecited by 664
  31. 2022PAL: Program-aided Language Modelscited by 652
  32. 2023A Comprehensive Overview of Large Language Modelscited by 648
  33. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  34. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  35. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  36. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  37. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  38. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578

50 most cited of 1,998 All 1,998 (JSON)