Knowledge Index of Noah's Ark
Not on the map. Why?
Abstract (arXiv)
Knowledge benchmarks for LLMs face three issues: scaling-driven designs that do not operationalize disciplinary representativeness; flat-payment annotation that permits lazy consensus; and unaudited ranking instability under bounded test budgets. We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines, with two formal results. First, we cast representativeness as a coverage-style objective over expert-elicited anchors and operationalize disciplinary representativeness through a proxy, yielding a (1-1/e) greedy approximation (Proposition 1); the guarantee applies to the proxy, not to population representativeness. Second, we prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality, with incentive-compatibility threshold B > Delta C / Delta p_min (Theorem 1). Evaluating 42 models from 13 labs, the top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%, leaving substantial headroom below saturation. The full leaderboard shows a tiered structure rather than a smooth total order: a small frontier tier lies above 48%, a dense strong-model tier spans roughly 38-45%, and low-performing models remain only modestly above the 10% chance baseline. Tool augmentation adds up to 5.17 points across the five tool-use evaluations, with gains varying substantially across models. We report bootstrap ranking-stability statistics to make bounded-budget variance explicit and to discourage over-interpretation of adjacent ranks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 3 Jun 2026.
Builds on
18 of 29 references.
References · 18
- 2025Qwen3 Technical Reportcited by 8,820
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
- 2018Datasheets for Datasetscited by 1,386
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
- 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
- 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
- 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
6 more
- 2025Humanity's Last Examcited by 491
- 2024Are We Done with MMLU?cited by 194
- 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 190
- 2026Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameterscited by 36
- 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
- 2026HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Examcited by 12
Built on by
Not cited yet.