# Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

arXiv: https://arxiv.org/abs/2604.24827 · HTML: https://mlsanity.com/paper/2604.24827/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 64 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Builds on (most cited first, 15 of 15)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [How much do language models memorize?](https://mlsanity.com/paper/2505.24832/) (2025)
- [Scalable Fingerprinting of Large Language Models](https://mlsanity.com/paper/2502.07760/) (2025)
- [Towards a Comprehensive Scaling Law of Mixture-of-Experts](https://mlsanity.com/paper/2509.23678/) (2025)
- [RAFP: Identifying LLM Lineages via Rare-Region Fingerprints](https://mlsanity.com/paper/2505.12682/) (2025)
- [Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications](https://mlsanity.com/paper/2602.16201/) (2026)
- [Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures](https://mlsanity.com/paper/2510.16968/) (2025)

## Built on by (most cited first, 17 of 17)

- [Mendel G"odel Machine: Recursive Self-Improving Coding Agents via Comparative Evolution](https://mlsanity.com/paper/2608.07645/) (2026)
- [AssayBench: An Assay-Level Virtual Cell Benchmark for LLMs and Agents](https://mlsanity.com/paper/2605.10876/) (2026)
- [KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing](https://mlsanity.com/paper/2605.29524/) (2026)
- [User as Engram: Internalizing Per-User Memory as Local Parametric Edits](https://mlsanity.com/paper/2606.19172/) (2026)
- [Inferring the Size of Large Language Models From Popular Text Memorization](https://mlsanity.com/paper/2605.29223/) (2026)
- [Building and Evaluating Fixed-Voice Thai TTS from Synthetic Speech](https://mlsanity.com/paper/2609.03502/) (2026)
- [The Grounding Gap: How LLMs Anchor the Meaning of Abstract Concepts Differently from Humans](https://mlsanity.com/paper/2605.08837/) (2026)
- [LLM-Metrics: Measuring Research Impact Through Large Language Model Memory](https://mlsanity.com/paper/2605.22176/) (2026)
- [Validity Threats for Foundation Model Research](https://mlsanity.com/paper/2606.05029/) (2026)
- [LEVANTE-bench: Multi-Scale Comparison of VLMs to Children Using Cognitive Tasks (or, "Is Your VLM Smarter Than a 5th Grader?")](https://mlsanity.com/paper/2606.05497/) (2026)
- [Black-Box Inference of LLM Architectural Properties with Restrictive API Access](https://mlsanity.com/paper/2607.01313/) (2026)
- [Practical Source Code Recovery from Binary Functions Using Anchor-Based Retrieval and LLM Reasoning](https://mlsanity.com/paper/2607.09452/) (2026)
- [The Model Knows Your Project, Not You: Measuring Recognition in LLMs with NameRank](https://mlsanity.com/paper/2607.12520/) (2026)
- [Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques](https://mlsanity.com/paper/2608.06609/) (2026)
- [From Caveman to Expert Analyst: Energy Consumption of Variable LLM Tasks](https://mlsanity.com/paper/2608.12350/) (2026)
- [AKRASIA: Stealthy Backdoor Attack on Reasoning-based Code LLMs](https://mlsanity.com/paper/2609.01023/) (2026)
- [SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC](https://mlsanity.com/paper/2609.18110/) (2026)

## Abstract (arXiv)

Closed-source frontier labs do not disclose parameter counts. Storing F facts requires at least F/(bits per parameter) weights, so factual recall lower-bounds parameter count--an intrinsic, serving-independent signal, though (as we show) a coarse one. We introduce Incompressible Knowledge Probes (IKPs), a benchmark of 1,400 factual questions spanning 7 tiers of obscurity, designed to isolate knowledge that cannot be derived by reasoning or compressed by architectural improvements. We score with no hallucination penalty (lambda = 0: IKP accuracy is simply the fraction of probed facts answered correctly), which removes both the penalty hyperparameter and the per-tier flooring choice; a full lambda x flooring ablation shows the calibration is robust across scoring choices while individual estimates are not, motivating the no-penalty default. We calibrate a log-linear mapping from IKP accuracy to parameter count on 93 open-weight models (135M-1,600B) spanning 19 vendors, achieving R^2 = 0.910; leave-one-out cross-validation confirms generalization (median fold error 1.48x, 72% within 2x, 86% within 3x). The instrument is deliberately coarse--its 90% prediction interval spans ~3x in either direction, wider than inference economics--so IKP recovers order-of-magnitude effective capacity and relative rankings, not precise parameter counts. For Mixture-of-Experts models, total parameters predict knowledge (R^2 = 0.67) better than active parameters (R^2 = 0.41). We evaluate 201 models from 27 vendors on a curated probe set (1,311 of 1,400 probes surviving name-collision and label-ambiguity filters) and report effective knowledge capacity for all major proprietary frontier models as prediction bands rather than point estimates; for heavily safety-tuned models these are lower bounds, since refusal policy can suppress tens of percentage points of otherwise-answerable capacity.