KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Many real-world questions appear deceptively simple yet implicitly demand two capabilities: (i) systematic coverage of a bounded knowledge universe and (ii) compositional set-based reasoning over that universe, a phenomenon we term "the tip of the iceberg." We formalize this challenge through two orthogonal dimensions: knowledge width, the cardinality of the required universe, and reasoning depth, the number of compositional set operations. We introduce KnowledgeBerg, a benchmark of 4,800 multiple-choice questions derived from 1,183 enumeration seeds spanning 10 domains and 17 languages, with universes grounded in authoritative sources to ensure reproducibility. Representative open-source LLMs demonstrate severe limitations, achieving only 5.26-36.88 F1 on universe enumeration and 16.00-44.19 accuracy on knowledge-grounded reasoning. Diagnostic analyses reveal three stages of failure: completeness, or missing knowledge; awareness, or failure to identify requirements; and application, or incorrect reasoning execution. This pattern persists across languages and model scales. Although test-time compute and retrieval augmentation yield measurable gains -- up to 4.35 and 3.78 points, respectively -- substantial gaps remain, exposing limitations in how current LLMs organize structured knowledge and execute compositional reasoning over bounded domains. The dataset is available at https://huggingface.co/datasets/2npc/KnowledgeBerg
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 40 references that are in the map. First posted on arXiv 19 Apr 2026.
Builds on
8 of 40 references.
Closest ancestors · 1
Same direction, newest first
- 2025Qwen3 Technical Report14 May
Other references · 7
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2025Gemma 3 Technical Reportcited by 1,824
- 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 147
- 2025Multidimensional Consistency Improves Reasoning in Language Modelscited by 6
- 2025OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Ontological Understanding, Reasoning and Learningcited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025When Reasoning Meets Its Laws
- 2025Follow the Path: Reasoning over Knowledge Graph Paths to Improve Large Language Model Factuality
- 2025OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases
- 2025Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
- 2024Hierarchical Deconstruction of LLM Reasoning: A Graph-Based Framework for Analyzing Knowledge Utilization
- 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
- 2024AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning
- 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models