mlsanityData to 7 Oct 2026

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

arXiv 2210.09261 ↗First posted 17 Oct 2022cs.CL cs.AIcited by 1,767

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models. Language models have already made good progress on this benchmark, with the best model in the BIG-Bench paper outperforming average reported human-rater results on 65% of the BIG-Bench tasks via few-shot prompting. But on what tasks do language models fall short of average human-rater performance, and are those tasks actually unsolvable by current language models? In this work, we focus on a suite of 23 challenging BIG-Bench tasks which we call BIG-Bench Hard (BBH). These are the task for which prior language model evaluations did not outperform the average human-rater. We find that applying chain-of-thought (CoT) prompting to BBH tasks enables PaLM to surpass the average human-rater performance on 10 of the 23 tasks, and Codex (code-davinci-002) to surpass the average human-rater performance on 17 of the 23 tasks. Since many tasks in BBH require multi-step reasoning, few-shot prompting without CoT, as done in the BIG-Bench evaluations (Srivastava et al., 2022), substantially underestimates the best performance and capabilities of language models, which is better captured via CoT prompting. As further analysis, we explore the interaction between CoT and model scale on BBH, finding that CoT enables emergent task performance on several BBH tasks with otherwise flat scaling curves.

Key facts. Cited by 1,767 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 67 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

49 of 67 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango16 Sep
  2. 2022Large Language Models are Zero-Shot Reasoners24 May
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
32 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  12. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  13. 2021MetaICL: Learning to Learn In Contextcited by 450
  14. 2021True Few-Shot Learning with Language Modelscited by 343
  15. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  16. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  17. 2021Scaling Laws for Transfercited by 265
  18. 2022Can language models learn from explanations in context?cited by 248
  19. 2022Binding Language Models in Symbolic Languagescited by 224
  20. 2022Predictability and Surprise in Large Generative Modelscited by 209
  21. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  22. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  23. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  24. 2021A Recipe For Arbitrary Text Style Transfer with Large Language Modelscited by 107
  25. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  26. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  27. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  28. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  29. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  30. 2022Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Modelscited by 49
  31. 2020Towards Interpretable Natural Language Understanding with Explanations as Latent Variablescited by 30
  32. 2022AmbiPun: Generating Humorous Puns with Ambiguous Contextcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 216; 2024: 460; 2025: 572; 2026: 499.

1,767 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2023Mistral 7Bcited by 3,065
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2024Qwen2 Technical Reportcited by 2,076
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2024Mixtral of Expertscited by 1,785
  11. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
38 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  8. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  10. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  12. 2023Teaching Large Language Models to Self-Debugcited by 963
  13. 2023BloombergGPT: A Large Language Model for Financecited by 923
  14. 2025Large Language Diffusion Modelscited by 872
  15. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  16. 2023Large Language Models as Optimizerscited by 830
  17. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  18. 2022Galactica: A Large Language Model for Sciencecited by 736
  19. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  20. 2024DeepSeek-V3 Technical Reportcited by 724
  21. 2022A Survey on In-context Learningcited by 717
  22. 2024Yi: Open Foundation Models by 01.AIcited by 706
  23. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  24. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  25. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  26. 2022PAL: Program-aided Language Modelscited by 635
  27. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  28. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  29. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  30. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  31. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  32. 2025Kimi K2: Open Agentic Intelligencecited by 523
  33. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  34. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  35. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  36. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  37. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  38. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405

50 most cited of 1,767 All 1,767 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
  2. 2025LIFEBench: Evaluating Length Instruction Following in Large Language Models
  3. 2023Stress Testing Chain-of-Thought Prompting for Large Language Models
  4. 2025Are Large Language Models the future crowd workers of Linguistics?
  5. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
  6. 2024The Impact of Reasoning Step Length on Large Language Models
  7. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods
  8. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision