Data to 14 Oct 2026

DoGE: Domain Reweighting with Generalization Estimation

arXiv 2310.15393 ↗First posted 23 Oct 2023cs.LG cs.AI cs.CLcited by 90

Not on the map. Why?

Abstract (arXiv)

The coverage and composition of the pretraining data significantly impacts the generalization ability of Large Language Models (LLMs). Despite its importance, recent LLMs still rely on heuristics and trial and error to increase or reduce the influence of data-domains. We propose DOmain reweighting with Generalization Estimation (DoGE), which optimizes the probability of sampling from each domain (domain weights) in a principled way. Our approach is a two-stage process consisting of (i) training a proxy model to obtain domain weights using a bi-level optimization algorithm; (ii) training a larger base model by sampling training domains according to the learned domain weights. In our experiments, we extensively show how DoGE improves the generalization of the base model to any target data mixture. On the SlimPajama dataset, our base model gets better perplexity and few-shot reasoning accuracies across 6 tasks compared to baseline methods. Moreover, aiming to generalize to out-of-domain target tasks, which is unseen in the pretraining corpus (OOD domain), DoGE can effectively identify inter-domain dependencies, and consistently achieves better test perplexity on the target domain.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Builds on 19 of its 41 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

19 of 41 references.

References · 19

2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2020Scaling Laws for Neural Language Modelscited by 6,586
2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
2017Crowdsourcing Multiple Choice Science Questionscited by 765
2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399

7 more

2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
2023Data Selection for Language Models via Importance Resamplingcited by 329
2019On The Power of Curriculum Learning in Training Deep Networkscited by 286
2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 182
2023Language acquisition: do children and language models follow similar learning stages?cited by 41
2022First is Better Than Last for Language Data Influencecited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 15; 2025: 39; 2026: 35.

90 papers.

Sort:

By area:

2025Qwen3 Technical Reportcited by 8,820
2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 76
2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 66
2025Scaling Laws for Optimal Data Mixturescited by 52
2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
2024Data Selection via Optimal Control for Language Modelscited by 38
2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 26
2024Balanced Data Sampling for Language Model Training with Clusteringcited by 23
2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19

38 more

2025Language Models Improve When Pretraining Data Matches Target Taskscited by 19
2023Data Management For Training Large Language Models: A Surveycited by 17
2025Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretrainingcited by 16
2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 14
2024Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-trainingcited by 12
2024Data, Data Everywhere: A Guide for Pretraining Dataset Constructioncited by 11
2026AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvementcited by 11
2024Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaborationcited by 10
2025SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversitycited by 10
2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
2025DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collectionscited by 9
2025PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflictscited by 8
2025IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignmentcited by 8
2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7
2025Graph Data Selection for Domain Adaptation: A Model-Free Approachcited by 7
2025Fast Data Attribution for Text-to-Image Modelscited by 7
2024Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasetscited by 6
2024Training Bilingual LMs with Data Constraints in the Targeted Languagecited by 6
2025DUET: Optimizing Training Data Mixtures via Feedback from Unseen Evaluation Taskscited by 6
2025TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-trainingcited by 6
2026Scaling Laws for Mixture Pretraining Under Data Constraintscited by 6
2025Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Modelscited by 5
2026DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Modelscited by 5
2026Can Generalist Agents Automate Data Curation?cited by 5
2025Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-trainingcited by 4
2025DIDS: Domain Impact-aware Data Sampling for Large Language Model Trainingcited by 4
2025A Survey of LLM × DATAcited by 4
2025GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretrainingcited by 4
2026DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixturescited by 4
2024VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMscited by 3
2025Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weightscited by 3
2025ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data Selectioncited by 3
2025R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Trainingcited by 3
2025LoReUn: Data Itself Implicitly Provides Cues to Improve Machine Unlearningcited by 3
2025BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretrainingcited by 3
2025Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practicecited by 3

50 most cited of 90 All 90 (JSON)