mlsanityData to 7 Oct 2026

Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?

arXiv 2608.10690 ↗First posted 11 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pretraining corpus composition shapes LLM capabilities, but it often remains hidden even when model weights are released. Prior work has inferred corpus mixtures or traced specific token groups from released tokenizer vocabularies; in contrast, we estimate corpus ratios for arbitrary target tokens. We first show that BPE tokenizers trained on different corpora share stable token ID--ratio distributions, motivating distribution transfer from known corpora to a target tokenizer trained on hidden corpora. We then propose Quantile-Guided Density Estimation (QGDE), which approximates this distribution with multiple quantile trends and uses local density weighting to produce token-level estimates. In controlled settings and a realistic setting using the released SmolLM tokenizer, QGDE achieves mean relative errors as low as 3.00% for token-level estimation and 3.08% after aggregation into category-level mixtures. These results suggest that released tokenizer vocabularies provide a useful signal for fine-grained corpus estimation beyond coarse composition inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

20 of 33 references.

References · 20

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  12. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
8 more
  1. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  2. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  3. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  4. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  5. 2020Fast WordPiece Tokenizationcited by 85
  6. 2023The Foundation Model Transparency Indexcited by 63
  7. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  8. 2025Speculating LLMs' Chinese Training Data Pollution from Their Tokenscited by 5

Built on by

Not cited yet.