# AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models

arXiv: https://arxiv.org/abs/2511.13029 · HTML: https://mlsanity.com/paper/2511.13029/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 17 Nov 2025.

## Builds on (most cited first, 10 of 10)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks](https://mlsanity.com/paper/2510.04374/) (2025)
- [Long-form factuality in large language models](https://mlsanity.com/paper/2403.18802/) (2024)
- [Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Models](https://mlsanity.com/paper/2410.08414/) (2024)

## Built on by (most cited first, 13 of 13)

- [Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2606.15007/) (2026)
- [WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution](https://mlsanity.com/paper/2608.27454/) (2026)
- [Hallucinations Undermine Trust; Metacognition is a Way Forward](https://mlsanity.com/paper/2605.01428/) (2026)
- [CausalGame: Benchmarking Causal Thinking of LLM Agents in Games](https://mlsanity.com/paper/2607.04293/) (2026)
- [Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models](https://mlsanity.com/paper/2607.08186/) (2026)
- [Unlocking Lossless Speedups in LLMs via Discrete Diffusion](https://mlsanity.com/paper/2609.04010/) (2026)
- [Toward Evaluation Frameworks for Multi-Agent Scientific AI Systems](https://mlsanity.com/paper/2603.26718/) (2026)
- [Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs](https://mlsanity.com/paper/2605.07806/) (2026)
- [Revealing Behavioral Plasticity in Large Language Models: A Token-Conditional Perspective](https://mlsanity.com/paper/2603.08398/) (2026)
- [Integrating Local and Global Entropy for Uncertainty Quantification in LLMs](https://mlsanity.com/paper/2606.09875/) (2026)
- [Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains](https://mlsanity.com/paper/2607.18438/) (2026)
- [Motif 3: Technical Report](https://mlsanity.com/paper/2608.09119/) (2026)
- [SingProbe Technical Report](https://mlsanity.com/paper/2608.30703/) (2026)

## Abstract (arXiv)

Existing language model evaluations primarily measure general capabilities, yet reliable use of these models across a range of domains demands factual accuracy and recognition of knowledge gaps. We introduce AA-Omniscience, a benchmark designed to measure both factual recall and knowledge calibration across 6,000 questions. Questions are derived from authoritative academic and industry sources, and cover 42 economically relevant topics within six different domains. The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall that jointly penalizes hallucinations and rewards abstention when uncertain, with 0 equating to a model that answers questions correctly as much as it does incorrectly. Among evaluated models, Claude 4.1 Opus attains the highest score (4.8), making it one of only three models to score above zero. These results reveal persistent factuality and calibration weaknesses across frontier models. Performance also varies by domain, with the models from three different research labs leading across the six domains. This performance variability suggests models should be chosen according to the demands of the use case rather than general performance for tasks where knowledge is important.