mlsanityData to 14 Oct 2026

GRIP: Geometric Refinement and Adaptive Information Potential for Data Efficiency

arXiv 2603.00031 ↗First posted 4 Feb 2026cs.CL cs.LGcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The performance of Large Language Models (LLMs) is increasingly governed by data efficiency rather than raw scaling volume. However, existing selection methods often decouple global distribution balancing from local instance selection, compromising the hierarchical integrity of the training set. We introduce GRIP (Geometric Refinement and Adaptive Information Potential), a framework that unifies these dimensions by modeling the corpus as an information-dense geometric space. GRIP employs a Rapid Adaptation Probe (RAP) to quantify the information potential of semantic clusters, dynamically re-allocating the sampling budget to regions with the highest representation deficits. Subsequently, we perform Intra-Cluster Selection using a length-rectified geometric prior to counteract embedding density artifacts and preserve long-tail logical sequences. Extensive evaluations on Mixture-of-Experts (MoE) models up to 300B tokens demonstrate that GRIP consistently outperforms state-of-the-art baselines, surpassing the performance of models trained on 3× larger uncurated datasets. Our work establishes a robust geometric foundation for adaptive data curation in large-scale pre-training.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 34 of its 40 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

34 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May

Other references · 32

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  9. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  10. 2023StarCoder: may the source be with you!cited by 799
  11. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  12. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
20 more
  1. 2023Textbooks Are All You Needcited by 569
  2. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  3. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  4. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  5. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  6. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  7. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  8. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  9. 2020A Theory of Usable Information Under Computational Constraintscited by 162
  10. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  11. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  12. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  13. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  14. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77
  15. 2024Capturing the Temporal Dependence of Training Data Influencecited by 35
  16. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  17. 2025D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuningcited by 21
  18. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  19. 2024Length-Induced Embedding Collapse in PLM-based Modelscited by 9
  20. 2025Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practicecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026UniGeM: Unifying Data Mixing and Selection via Geometric Exploration and Mining
  2. 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression
  3. 2025Multi-Scale Manifold Alignment for Interpreting Large Language Models: A Unified Information-Geometric Framework
  4. 2026GEM: Geometric Entropy Mixing for Optimal LLM Data Curation
  5. 2026GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models
  6. 2025On the Fundamental Limits of LLMs at Scale
  7. 2025Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
  8. 2026A Geometric Perspective on Next-Token Prediction in Large Language Models: Three Emerging Phases