mlsanityData to 7 Oct 2026

Self-Influence Guided Data Reweighting for Language Model Pre-training

arXiv 2311.00913 ↗First posted 2 Nov 2023cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Language Models (LMs) pre-trained with self-supervision on large text corpora have become the default starting point for developing models for various NLP tasks. Once the pre-training corpus has been assembled, all data samples in the corpus are treated with equal importance during LM pre-training. However, due to varying levels of relevance and quality of data, equal importance to all the data samples may not be the optimal choice. While data reweighting has been explored in the context of task-specific supervised learning and LM fine-tuning, model-driven reweighting for pre-training data has not been explored. We fill this important gap and propose PRESENCE, a method for jointly reweighting samples by leveraging self-influence (SI) scores as an indicator of sample importance and pre-training. PRESENCE promotes novelty and stability for model pre-training. Through extensive analysis spanning multiple model sizes, datasets, and tasks, we present PRESENCE as an important first step in the research direction of sample reweighting for pre-training language models.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

39 of 49 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  9. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
27 more
  1. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  2. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  3. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  4. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  5. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  6. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  7. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  8. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  9. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 350
  10. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  11. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 259
  12. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  13. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  14. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  15. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  16. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  17. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  18. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  19. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 52
  20. 2019Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimizationcited by 51
  21. 2019Cloze-driven Pretraining of Self-attention Networkscited by 41
  22. 2020Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedupcited by 31
  23. 2022Mitigating Dataset Bias by Using Per-sample Gradientcited by 21
  24. 2023Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasetscited by 18
  25. 2019Scale MLPerf-0.6 models on Google TPU-v3 Podscited by 14
  26. 2021Memory Augmented Optimizers for Deep Learningcited by 5
  27. 2022Selective classification using a robust meta-learning approachcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 17; 2026: 5.

37 papers.

Sort:

By area:

  1. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  2. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  3. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  4. 2024Data Selection via Optimal Control for Language Modelscited by 38
  5. 2024ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweightingcited by 35
  6. 2023Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answeringcited by 28
  7. 2025How to Train Your LLM Web Agent: A Statistical Diagnosiscited by 18
  8. 2025Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretrainingcited by 16
  9. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  10. 2024Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Modelscited by 10
  11. 2024Semantic Self-Consistency: Enhancing Language Model Reasoning via Semantic Weightingcited by 9
  12. 2025Enhancing Training Data Attribution with Representational Optimizationcited by 8
25 more
  1. 2024LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectivescited by 6
  2. 2025First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimationcited by 5
  3. 2025FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategycited by 4
  4. 2025Disentangling the Roles of Representation and Selection in Data Pruningcited by 4
  5. 2026DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Modelscited by 4
  6. 2025Unraveling Indirect In-Context Learning Using Influence Functionscited by 3
  7. 2025Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weightscited by 3
  8. 2025ToReMi: Topic-Aware Data Reweighting for Dynamic Pre-Training Data Selectioncited by 3
  9. 2025Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understandingcited by 3
  10. 2026Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methodscited by 3
  11. 2024The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcarecited by 2
  12. 2025Enhancing LLMs via High-Knowledge Data Selectioncited by 2
  13. 2024ProgGen: Generating Named Entity Recognition Datasets Step-by-step with Self-Reflexive Large Language Modelscited by 1
  14. 2024Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiencescited by 1
  15. 2024Channel Matters: Estimating Channel Influence for Multivariate Time Seriescited by 1
  16. 2024Variational Model Merging for Pareto Front Estimation in Multitask Finetuningcited by 1
  17. 2025RELexED: Retrieval-Enhanced Legal Summarization with Exemplar Diversitycited by 1
  18. 2025FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Trainingcited by 1
  19. 2024EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Modelscited by 0
  20. 2025Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretrainingcited by 0
  21. 2025What Is The Performance Ceiling of My Classifier? Utilizing Category-Wise Influence Functions for Pareto Frontier Analysiscited by 0
  22. 2025Explore and Establish Synergistic Effects Between Weight Pruning and Coreset Selection in Neural Network Trainingcited by 0
  23. 2026Quantifying the Agreement Between Data-Influence and Data-Similarity to Understand LLM Behaviorcited by 0
  24. 2026Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuningcited by 0
  25. 2026Invent a Dataset: Measuring dataset generation abilities with zero seedcited by 0