# Token Counts Are Not Model Lineage: A Frozen-Threshold Holdout Study of Black-Box LLM API Fingerprinting

arXiv: https://arxiv.org/abs/2608.29930 · HTML: https://mlsanity.com/paper/2608.29930/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 30 Aug 2026.

## Builds on (most cited first, 7 of 7)

- [A Fingerprint for Large Language Models](https://mlsanity.com/paper/2407.01235/) (2024)
- [Hide and Seek: Fingerprinting Large Language Models with Evolutionary Learning](https://mlsanity.com/paper/2408.02871/) (2024)
- [ProFLingo: A Fingerprinting-based Intellectual Property Protection Scheme for Large Language Models](https://mlsanity.com/paper/2405.02466/) (2024)
- [Fingerprinting LLMs via Prompt Injection](https://mlsanity.com/paper/2509.25448/) (2025)
- [The Challenge of Identifying the Origin of Black-Box Large Language Models](https://mlsanity.com/paper/2503.04332/) (2025)
- [KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing](https://mlsanity.com/paper/2605.29524/) (2026)
- [Black-Box Forensics for Conversational LLM Agents](https://mlsanity.com/paper/2606.22698/) (2026)

## Abstract (arXiv)

Black-box model attribution is increasingly relevant when large language models (LLMs) are served through relay and reseller APIs. A tempting low-cost signal is the prompt-token count returned by an OpenAI-compatible endpoint: two models that share a tokenizer and chat template may produce the same count sequence up to a fixed offset. Yet the validity of this signal for broader model-family attribution has received little direct holdout testing. We conduct a frozen-threshold study over 24 labeled endpoint pairs, split evenly into a development set and an untouched holdout set, with three temporal repeats and 30 controlled texts per pair. We introduce a validity-gated result contract that distinguishes an observed dissimilarity from an uninformative measurement caused by missing usage data, rate limits, or endpoint policy. The resulting shift-invariant exact-match score perfectly separates the 12 development pairs, yielding a frozen threshold of 0.725. On holdout, however, only 6 of 12 pairs are eligible under the pre-specified three-repeat rule. Among eligible pairs, balanced accuracy is 0.75, sensitivity is 0.50 (95% Wilson interval 0.15--0.85), and specificity is 1.00 (0.342--1.00). Two same-family pairs---Qwen 3.8 and DeepSeek V4 variants---fall below the frozen threshold. Across 4,320 formal API calls, every log is replayable, while holdout contains 189 non-200 responses and 157 successful responses without prompt-token usage. The study therefore validates token-count consistency as a fingerprint of a shared tokenization stack, but rejects its use as a standalone necessary test for model-family lineage.