mlsanityData to 13 Oct 2026

HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

arXiv 2607.02266 ↗First posted 2 Jul 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Most data-mixing methods assume the corpus has already been partitioned into groups, and the choice of those groups determines what a mixer can express. Existing labels, including provenance, topic or format taxonomies, and flat embedding clusters, commit to one semantic axis at one granularity; changing the resolution rebuilds the labels. We argue the bottleneck is the label system, not the mixer, and provide a hierarchical one. HERMES is a data-derived labeling substrate: a Learned Semantic Transform followed by 3-stage residual vector quantization annotates each document once into a coarse-to-fine code whose prefix length controls granularity up to approximately 130k cells. At coarse granularity HERMES sits at a plateau with KMeans-family methods on standard clustering metrics, so the contribution is the substrate, not the clusterer. On 1B-parameter, 25B-token pre-training, the hierarchy exposes an interaction fixed-granularity pipelines cannot test: at one prefix length, a combined Stage-2 rule contrast, equal-subbucket coverage versus size-proportional within-bucket quality top-30%, lifts a 16-task capability macro-average by +0.0253; at the next finer level, the same rule loses its measurable edge as candidate pools contract approximately 5x. HERMES reframes data mixture design from choosing among fixed label sets to navigating a reusable, data-derived granularity hierarchy.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

30 of 41 references.

References · 30

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  2. 2020Scaling Laws for Neural Language Modelscited by 6,494
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  4. 2017Neural Discrete Representation Learningcited by 4,680
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,805
  8. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,242
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,126
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 762
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 760
  12. 2017Crowdsourcing Multiple Choice Science Questionscited by 751
18 more
  1. 2022Autoregressive Image Generation using Residual Quantizationcited by 709
  2. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 518
  3. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 391
  4. 2023Data Selection for Language Models via Importance Resamplingcited by 326
  5. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 182
  6. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 175
  7. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  8. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 145
  9. 2024OLMES: A Standard for Language Model Evaluationscited by 101
  10. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 98
  11. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 75
  12. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 65
  13. 2024Residual Quantization with Implicit Neural Codebookscited by 25
  14. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  15. 2025Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuningcited by 14
  16. 2015Improved Residual Vector Quantization for High-dimensional Approximate Nearest Neighbor Searchcited by 11
  17. 2025SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversitycited by 10
  18. 2025Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-trainingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning-Based Refinement of Unsupervised Text Clusters with LLMs
  2. 2025ClusterFusion: Hybrid Clustering with Embedding Guidance and LLM Adaptation
  3. 2026Quantize by Drift: Label-Free Mixed-Precision Post-Training Quantization for Text Embedders
  4. 2025Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixtures
  5. 2025ESMC: MLLM-Based Embedding Selection for Explainable Multiple Clustering
  6. 2024Scaling Laws for Fine-Grained Mixture of Experts
  7. 2026Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization
  8. 2026Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality