mlsanityData to 14 Oct 2026

BIG-Bench Extra Hard

arXiv 2502.19187 ↗First posted 26 Feb 2025cs.CLcited by 125

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in everyday applications, demanding robust general reasoning capabilities and diverse reasoning skillset. However, current LLM reasoning benchmarks predominantly focus on mathematical and coding abilities, leaving a gap in evaluating broader reasoning proficiencies. One particular exception is the BIG-Bench dataset, which has served as a crucial benchmark for evaluating the general reasoning capabilities of LLMs, thanks to its diverse set of challenging tasks that allowed for a comprehensive assessment of general reasoning across various skills within a unified framework. However, recent advances in LLMs have led to saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH). State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus diminishing its utility. To address this limitation, we introduce BIG-Bench Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM reasoning evaluation. BBEH replaces each task in BBH with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty. We evaluate various models on BBEH and observe a (harmonic) average accuracy of 9.8% for the best general-purpose model and 44.8% for the best reasoning-specialized model, indicating substantial room for improvement and highlighting the ongoing challenge of achieving robust general reasoning in LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 52 of its 66 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

52 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries19 Sep
  2. 2024Linguini: A benchmark for language-agnostic linguistic reasoning18 Sep
  3. 2024Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzles16 Sep
  4. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models23 Apr
  5. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information13 Jun
  6. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples24 May
  7. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct

Other references · 45

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
33 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  5. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  6. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
  7. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  8. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  9. 2021Time-Aware Language Models as Temporal Knowledge Basescited by 296
  10. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  11. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  12. 2024Large Language Models Can Learn Temporal Reasoningcited by 111
  13. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 108
  14. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  15. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 88
  16. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  17. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 77
  18. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 68
  19. 2023Evaluating Spatial Understanding of Large Language Modelscited by 66
  20. 2017A Large Self-Annotated Corpus for Sarcasmcited by 64
  21. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
  22. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 30
  23. 2024Not All LLM Reasoners Are Created Equalcited by 30
  24. 2023CRAB: Assessing the Strength of Causal Relationships Between Real-world Eventscited by 28
  25. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 19
  26. 2024Exploring and Benchmarking the Planning Capabilities of Large Language Modelscited by 19
  27. 2024Transformers Struggle to Learn to Searchcited by 19
  28. 2024SportQA: A Benchmark for Sports Understanding in Large Language Modelscited by 15
  29. 2013Graded Causation and Defaultscited by 13
  30. 2024Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problemscited by 7
  31. 2024Sports Intelligence: Assessing the Sports Understanding Capabilities of Language Models through Question Answering from Text to Videocited by 6
  32. 2024Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Modelscited by 4
  33. 2025Peeking Behind Closed Doors: Risks of LLM Evaluation by Private Data Curatorscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 59; 2026: 66.

125 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,824
  2. 2026Gemma 4 Technical Reportcited by 259
  3. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  4. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  5. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  6. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 75
  7. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
  8. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 54
  9. 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyondcited by 50
  10. 2025Inverse Scaling in Test-Time Computecited by 50
  11. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 50
  12. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
38 more
  1. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
  2. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 20
  3. 2025InternBootcamp: Boosting LLM Reasoning with Verifiable Task Scalingcited by 20
  4. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 19
  5. 2025ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shapingcited by 17
  6. 2025T5Gemma 2: Seeing, Reading, and Understanding Longercited by 17
  7. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  8. 2025The Serial Scaling Hypothesiscited by 13
  9. 2025HoneyBee: Data Recipes for Vision-Language Reasonerscited by 13
  10. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimizationcited by 13
  11. 2026ERNIE 5.0 Technical Reportcited by 11
  12. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 11
  13. 2025When Do Large Language Models Exhibit Unsolicited Deception?cited by 10
  14. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  15. 2026R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Trainingcited by 10
  16. 2025HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputscited by 9
  17. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  18. 2025Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBenchcited by 8
  19. 2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradientcited by 8
  20. 2025Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesiscited by 8
  21. 2025PretrainZero: Reinforcement Active Pretrainingcited by 8
  22. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  23. 2026Understanding the Challenges in Iterative Generative Optimization with LLMscited by 8
  24. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 7
  25. 2025Position: Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests insteadcited by 7
  26. 2025Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learningcited by 7
  27. 2025OpenSIR: Open-Ended Self-Improving Reasonercited by 7
  28. 2026Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debatecited by 7
  29. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  30. 2025SLR: Automated Synthesis for Scalable Logical Reasoningcited by 6
  31. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Modelscited by 6
  32. 2026Optimsyn: Influence-Guided Rubrics Optimization for Synthetic Data Generationcited by 6
  33. 2026Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scalecited by 6
  34. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  35. 2025Zero Reinforcement Learning Towards General Domainscited by 5
  36. 2026EvoMAS: Evolutionary Generation of Multi-Agent Systemscited by 5
  37. 2026Improving Interactive In-Context Learning from Natural Language Feedbackcited by 5
  38. 2026ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Modelscited by 5

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
  2. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
  3. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Training
  4. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  5. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models
  6. 2025Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models ↗
  7. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
  8. 2024Are Your LLMs Capable of Stable Reasoning?