mlsanityData to 7 Oct 2026

The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora

arXiv 2608.09093 ↗First posted 10 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

How a document's arrangement is written down, its notation, is a training variable that no dataset card records. The field has established that text-extraction choices change model behaviour, and has never once measured the notation of what those choices put into the corpus. We define clean-window survival, a deterministic count of how much of a stream still demands the boundary inference, and measure notation on three fronts. What corpora carry: a census of thirteen public corpora, where survival falls to 0.153 in a vision-converted PDF slice against 0.889 in C4; the scarce resource is not unmarked text but long unmarked text; a pre-registered supply test finds what remains institutional, not consumer. Our own pre-registered prediction failed: converters do not fabricate structure on prose, and that null forced the reliability mechanism that survives it. What readers use: across five base models spanning 0.6B to 8.2B and two pipelines, deleting a structural announcement makes the following prose measurably harder to predict, while swapping its notation moves nothing. That zero does not make notation unimportant; it relocates the variable: the operative cue is the announcement, not the sigil. What writers impose: a bounded null. Base models do not impose the marked register above the authored baseline, and handed prose with every announcement deleted they do not put one back, at a rate indistinguishable from zero against an authored reference of zero. We ship the format those measurements imply: the pure frame, paragraphs in authored order, every announcement deleted into a reversible sidecar, mixed against the marked copy over announcement presence rather than notation. Choose format operators by the capability they train, not by the fidelity they preserve, and record extractor identity and survival on data cards.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 77 of its 91 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

77 of 91 references.

References · 77

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  10. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  11. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  12. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
38 more
  1. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  2. 2024MinerU: An Open-Source Solution for Precise Document Content Extractioncited by 255
  3. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  4. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  5. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  6. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  7. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  8. 2023What's In My Big Data?cited by 129
  9. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 120
  10. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 90
  11. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 87
  12. 2023On the Foundations of Shortcut Learningcited by 64
  13. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  14. 2024Do language models plan ahead for future tokens?cited by 47
  15. 2025The Price of Format: Diversity Collapse in LLMscited by 43
  16. 2022Datasheet for the Pilecited by 41
  17. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 34
  18. 2025Idiosyncrasies in Large Language Modelscited by 32
  19. 2024Docling Technical Reportcited by 29
  20. 2025The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Textcited by 25
  21. 2024AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filterscited by 21
  22. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 19
  23. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingcited by 19
  24. 2022ChapterBreak: A Challenge Dataset for Long-Range Language Modelscited by 18
  25. 2019Mutual Information Scaling and Expressive Power of Sequence Modelscited by 14
  26. 2025LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMscited by 11
  27. 2024The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generationcited by 9
  28. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  29. 2025Dripper: Token-Efficient Main HTML Extraction with a Lightweight LMcited by 8
  30. 2023Autocorrelations Decay in Texts and Applicability Limits of Language Modelscited by 7
  31. 2025Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usabilitycited by 7
  32. 2025Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variationcited by 6
  33. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Datacited by 6
  34. 2024Document Structure in Long Document Transformerscited by 5
  35. 2025Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountabilitycited by 5
  36. 2025EntropyLong: Effective Long-Context Training via Predictive Uncertaintycited by 5
  37. 2025Predicting the Emergence of Induction Heads in Language Model Pretrainingcited by 5
  38. 2026propella-1: Multi-Property Document Annotation for LLM Data Curation at Scalecited by 5

50 most cited of 77 All 77 (JSON)

Built on by

Not cited yet.