mlsanityData to 14 Oct 2026

DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models

arXiv 2603.23514 ↗First posted 5 Mar 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models appear competent when answering general questions but often fail when pushed into domain-specific details. No existing methodology provides an out-of-the-box solution for measuring how deeply LLMs can sustain accurate responses under adaptive follow-up questioning across arbitrary domains. We present DepthCharge, a domain-agnostic framework that measures knowledge depth through three innovations: adaptive probing that generates follow-up questions based on concepts the model actually mentions, on-demand fact verification from authoritative sources, and survival statistics with constant sample sizes at every depth level. The framework can be deployed on any knowledge domain with publicly verifiable facts, without requiring pre-constructed test sets or domain-specific expertise. DepthCharge results are relative to the evaluator model used for answer checking, making the framework a tool for comparative evaluation rather than absolute accuracy certification. Empirical validation across four diverse domains (Medicine, Constitutional Law, Ancient Rome, and Quantum Computing) with five frontier models demonstrates that DepthCharge reveals depth-dependent performance variation hidden by standard benchmarks. Expected Valid Depth (EVD) ranges from 3.45 to 7.55 across model-domain combinations, and model rankings vary substantially by domain, with no single model dominating all areas. Cost-performance analysis further reveals that expensive models do not always achieve deeper knowledge, suggesting that domain-specific evaluation is more informative than aggregate benchmarks for model selection in professional applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 5 Mar 2026.

Builds on

38 of 45 references.

References · 38

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  6. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  7. 2025Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Modelscited by 62
  8. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  9. 2025Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theorycited by 45
  10. 2025Evaluating Large Language Models in Scientific Discoverycited by 42
  11. 2025IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theorycited by 40
  12. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
26 more
  1. 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 38
  2. 2025A Survey on Data Contamination for Large Language Modelscited by 35
  3. 2025LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validationcited by 34
  4. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 31
  5. 2025A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domainscited by 28
  6. 2025Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarkscited by 28
  7. 2025Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Taskscited by 17
  8. 2025Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Searchcited by 10
  9. 2025Learning Compact Representations of LLM Abilities via Item Response Theorycited by 10
  10. 2025OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarkingcited by 9
  11. 2025Pitfalls of Evaluating Language Models with Open Benchmarkscited by 7
  12. 2025BloomAPR: A Bloom's Taxonomy-based Framework for Assessing the Capabilities of LLM-Powered APR Solutionscited by 5
  13. 2025Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Lengthcited by 5
  14. 2025Aligning Knowledge Graphs and Language Models for Factual Accuracycited by 4
  15. 2025Automatic Question & Answer Generation Using Generative Large Language Model (LLM)cited by 4
  16. 2025LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluationcited by 4
  17. 2024BloomWise: Enhancing Problem-Solving capabilities of Large Language Models using Bloom's-Taxonomy-Inspired Promptscited by 3
  18. 2025Evaluating LLM-Generated Q&A Test: a Student-Centered Studycited by 3
  19. 2025Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solvingcited by 3
  20. 2025Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judgescited by 3
  21. 2025Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddingscited by 3
  22. 2025MedKGEval: A Knowledge Graph-Based Multi-Turn Evaluation Framework for Open-Ended Patient Interactions with Clinical LLMscited by 3
  23. 2025Improving Factuality in LLMs via Inference-Time Knowledge Graph Constructioncited by 2
  24. 2025Towards Unification of Hallucination Detection and Fact Verification for Large Language Modelscited by 2
  25. 2025Testing Question Answering Software with Context-Driven Question Generationcited by 1
  26. 2025Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Modelscited by 1

Built on by

Not cited yet.