mlsanityData to 7 Oct 2026

UniGeM: Unifying Data Mixing and Selection via Geometric Exploration and Mining

arXiv 2602.03772 ↗First posted 3 Feb 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The scaling of Large Language Models (LLMs) is increasingly limited by data quality. Most methods handle data mixing and sample selection separately, which can break the structure in code corpora. We introduce UniGeM, a framework that unifies mixing and selection by treating data curation as a textitmanifold approximation problem without training proxy models or relying on external reference datasets. UniGeM operates hierarchically: Macro-Exploration learns mixing weights with stability-based clustering; Micro-Mining filters high-quality instances by their geometric distribution to ensure logical consistency. Validated by training 8B and 16B MoE models on 100B tokens, UniGeM achieves 2.0× data efficiency over a random baseline and further improves overall performance compared to SOTA methods in reasoning-heavy evaluations and multilingual generalization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

21 of 32 references.

References · 21

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  9. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  10. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  11. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  12. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
9 more
  1. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  2. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  3. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  4. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  5. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  6. 2021Learning Signal-Agnostic Manifolds of Neural Fieldscited by 45
  7. 2024Meta Flow Matching: Integrating Vector Fields on the Wasserstein Manifoldcited by 41
  8. 2024ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Lawscited by 15
  9. 2025Dataset Distillation as Pushforward Optimal Quantizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
  2. 2026Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality
  3. 2026GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency
  4. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models
  5. 2026Aligning the Query Space: Greedy Information Projection for Language Model Data Selection
  6. 2026Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods
  7. 2024How to Train Data-Efficient LLMs
  8. 2026From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignment