# Token-Efficient Change Detection in LLM APIs

arXiv: https://arxiv.org/abs/2602.11083 · HTML: https://mlsanity.com/paper/2602.11083/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Builds on (most cited first, 18 of 18)

- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Towards an Understanding of Large Language Models in Software Engineering Tasks](https://mlsanity.com/paper/2308.11396/) (2023)
- [Discrete Distribution Estimation under Local Privacy](https://mlsanity.com/paper/1602.07387/) (2016)
- [LLMmap: Fingerprinting For Large Language Models](https://mlsanity.com/paper/2407.15847/) (2024)
- [Model Equality Testing: Which Model Is This API Serving?](https://mlsanity.com/paper/2410.20247/) (2024)
- [Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs](https://mlsanity.com/paper/2504.04715/) (2025)
- [TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification](https://mlsanity.com/paper/2402.12991/) (2024)
- [Differentially Private Testing of Identity and Closeness of Discrete Distributions](https://mlsanity.com/paper/1707.05128/) (2017)
- [Did the Model Change? Efficiently Assessing Machine Learning API Shifts](https://mlsanity.com/paper/2107.14203/) (2021)
- [Hypothesis Testing For Densities and High-Dimensional Multinomials: Sharp Local Minimax Rates](https://mlsanity.com/paper/1706.10003/) (2017)
- [Locally private non-asymptotic testing of discrete distributions is faster using interactive mechanisms](https://mlsanity.com/paper/2005.12601/) (2020)
- [Log Probability Tracking of LLM APIs](https://mlsanity.com/paper/2512.03816/) (2025)
- [Estimating Entropy of Distributions in Constant Space](https://mlsanity.com/paper/1911.07976/) (2019)
- [Hardware and Software Platform Inference](https://mlsanity.com/paper/2411.05197/) (2024)
- [Optimal Algorithms for Augmented Testing of Discrete Distributions](https://mlsanity.com/paper/2412.00974/) (2024)
- [Multinomial Goodness-of-Fit Based on U-Statistics: High-Dimensional Asymptotic and Minimax Optimality](https://mlsanity.com/paper/1812.08924/) (2018)
- [Testing High-dimensional Multinomials with Applications to Text Analysis](https://mlsanity.com/paper/2301.01381/) (2023)
- [Estimation of discrete distributions with high probability under chi2-divergence](https://mlsanity.com/paper/2510.25400/) (2025)

## Built on by (most cited first, 5 of 5)

- [Behavioral Fingerprints for LLM Endpoint Stability and Identity](https://mlsanity.com/paper/2603.19022/) (2026)
- [KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing](https://mlsanity.com/paper/2605.29524/) (2026)
- [Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs](https://mlsanity.com/paper/2608.16391/) (2026)
- [Strategic Self-Consistency](https://mlsanity.com/paper/2609.30352/) (2026)
- [Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting](https://mlsanity.com/paper/2610.17399/) (2026)

## Abstract (arXiv)

Remote change detection in LLMs is a difficult problem. Existing methods are either too expensive for deployment at scale, or require initial white-box access to model weights or grey-box access to log probabilities. We aim to achieve both low cost and strict black-box operation, observing only output tokens. Our approach hinges on specific inputs we call Border Inputs, for which there exists more than one output top token. From a statistical perspective, optimal change detection depends on the model's Jacobian and the Fisher information of the output distribution. Analyzing these quantities in low-temperature regimes shows that border inputs enable powerful change detection tests. Building on this insight, we propose the Black-Box Border Input Tracking (B3IT) scheme. Extensive in-vivo and in-vitro experiments show that border inputs are easily found for non-reasoning tested endpoints, and achieve performance on par with the best available grey-box approaches. B3IT reduces costs by 30× compared to existing methods, while operating in a strict black-box setting.