mlsanityData to 14 Oct 2026

GIM: Evaluating models via tasks that integrate multiple cognitive domains

arXiv 2605.18663 ↗First posted 18 May 2026cs.AI cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

As LLM benchmarks saturate, the evaluation community has pursued two strategies to increase difficulty: escalating knowledge demands (GPQA, HLE) or removing knowledge entirely in favor of abstract reasoning (ARC-AGI). The first conflates memorization with capability; the second divorces reasoning from the practical contexts in which it matters. We take a different approach. The Grounded Integration Measure (GIM) is a benchmark of 820 original problems (615 public, 205 private) where difficulty comes from integration; individual problems require coordinating multiple cognitive operations (constraint satisfaction, state tracking, epistemic vigilance, audience calibration) over broadly accessible knowledge, so that reasoning stays grounded in realistic tasks without being gated on specialized expertise. Each problem is an original expert-authored composition, majority with rubric-decomposed scoring. We calibrate a judge-aware continuous response 2-parameter logistic (2PL) IRT model across 53 test-configurations (unique model x thinking-level pairs) and five calibrated judges, using 203,800 epoch-averaged prompt-judge cells derived from >1M raw judge-scored observations, producing robust ability estimates that correctly order test-configurations even when raw accuracy is distorted by errors, missing data, or judge leniency differences. Using this framework, we present a comprehensive leaderboard spanning 22 models and 47 reporting test-configurations, and conduct what is to our knowledge the most extensive published study of how test-time compute trades off against model capability on a fixed benchmark: 11 models swept across 35 test-configurations. We observe that within-family configuration choices, such as thinking budget and quantization, matter as much as model selection, and increasing thinking tokens has diminishing marginal returns.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 42 of its 78 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

42 of 78 references.

References · 42

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  11. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  12. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
30 more
  1. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  5. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  8. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  10. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  11. 2025Humanity's Last Examcited by 491
  12. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  13. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  14. 2024Measuring short-form factuality in large language modelscited by 373
  15. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  16. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  17. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  18. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  19. 2019On the Measure of Intelligencecited by 144
  20. 2021The Benchmark Lotterycited by 136
  21. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 85
  22. 2021Comparing Test Sets with Item Response Theorycited by 57
  23. 2024The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learningcited by 41
  24. 2026ARC-AGI-3: A New Challenge for Frontier Agentic Intelligencecited by 37
  25. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  26. 2025CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmarkcited by 10
  27. 2025WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarioscited by 9
  28. 2025Understanding Reinforcement Learning for Model Training, and future directions with GRAPEcited by 2
  29. 2016Building an Evaluation Scale using Item Response Theorycited by 1
  30. 2026ZPD Detector: Data Selection via Capability-Difficulty Alignment for Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Assessing LLM Reasoning Steps via Principal Knowledge Grounding
  2. 2025Cognitive Foundations for Reasoning and Their Manifestation in LLMs
  3. 2025Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLM
  4. 2026Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation
  5. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
  6. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
  7. 2026Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
  8. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty