# DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models

arXiv: https://arxiv.org/abs/2603.23514 · HTML: https://mlsanity.com/paper/2603.23514/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 45 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 30 of 38)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Models](https://mlsanity.com/paper/2504.04717/) (2025)
- [Large Language Models Hallucination: A Comprehensive Survey](https://mlsanity.com/paper/2510.06265/) (2025)
- [Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory](https://mlsanity.com/paper/2505.15055/) (2025)
- [Evaluating Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2512.15567/) (2025)
- [IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory](https://mlsanity.com/paper/2506.01048/) (2025)
- [Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation](https://mlsanity.com/paper/2502.17521/) (2025)
- [Medical Large Language Model Benchmarks Should Prioritize Construct Validity](https://mlsanity.com/paper/2503.10694/) (2025)
- [A Survey on Data Contamination for Large Language Models](https://mlsanity.com/paper/2502.14425/) (2025)
- [LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation](https://mlsanity.com/paper/2506.04078/) (2025)
- [Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities](https://mlsanity.com/paper/2505.20099/) (2025)
- [A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domains](https://mlsanity.com/paper/2507.23486/) (2025)
- [Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks](https://mlsanity.com/paper/2511.04689/) (2025)
- [Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks](https://mlsanity.com/paper/2505.08903/) (2025)
- [Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search](https://mlsanity.com/paper/2502.01609/) (2025)
- [Learning Compact Representations of LLM Abilities via Item Response Theory](https://mlsanity.com/paper/2510.00844/) (2025)
- [OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking](https://mlsanity.com/paper/2511.08598/) (2025)
- [Pitfalls of Evaluating Language Models with Open Benchmarks](https://mlsanity.com/paper/2507.00460/) (2025)
- [BloomAPR: A Bloom's Taxonomy-based Framework for Assessing the Capabilities of LLM-Powered APR Solutions](https://mlsanity.com/paper/2509.25465/) (2025)
- [Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length](https://mlsanity.com/paper/2512.07019/) (2025)
- [Aligning Knowledge Graphs and Language Models for Factual Accuracy](https://mlsanity.com/paper/2507.13411/) (2025)
- [Automatic Question & Answer Generation Using Generative Large Language Model (LLM)](https://mlsanity.com/paper/2508.19475/) (2025)
- [LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluation](https://mlsanity.com/paper/2511.06346/) (2025)
- [BloomWise: Enhancing Problem-Solving capabilities of Large Language Models using Bloom's-Taxonomy-Inspired Prompts](https://mlsanity.com/paper/2410.04094/) (2024)
- [Evaluating LLM-Generated Q&A Test: a Student-Centered Study](https://mlsanity.com/paper/2505.06591/) (2025)

## Abstract (arXiv)

Large Language Models appear competent when answering general questions but often fail when pushed into domain-specific details. No existing methodology provides an out-of-the-box solution for measuring how deeply LLMs can sustain accurate responses under adaptive follow-up questioning across arbitrary domains. We present DepthCharge, a domain-agnostic framework that measures knowledge depth through three innovations: adaptive probing that generates follow-up questions based on concepts the model actually mentions, on-demand fact verification from authoritative sources, and survival statistics with constant sample sizes at every depth level. The framework can be deployed on any knowledge domain with publicly verifiable facts, without requiring pre-constructed test sets or domain-specific expertise. DepthCharge results are relative to the evaluator model used for answer checking, making the framework a tool for comparative evaluation rather than absolute accuracy certification. Empirical validation across four diverse domains (Medicine, Constitutional Law, Ancient Rome, and Quantum Computing) with five frontier models demonstrates that DepthCharge reveals depth-dependent performance variation hidden by standard benchmarks. Expected Valid Depth (EVD) ranges from 3.45 to 7.55 across model-domain combinations, and model rankings vary substantially by domain, with no single model dominating all areas. Cost-performance analysis further reveals that expensive models do not always achieve deeper knowledge, suggesting that domain-specific evaluation is more informative than aggregate benchmarks for model selection in professional applications.