mlsanityData to 14 Oct 2026

Disentangling the Roles of Representation and Selection in Data Pruning

arXiv 2507.03648 ↗First posted 4 Jul 2025cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Data pruning, selecting small but impactful subsets, offers a promising way to efficiently scale NLP model training. However, existing methods often involve many different design choices, which have not been systematically studied. This limits future developments. In this work, we decompose data pruning into two key components: the data representation and the selection algorithm, and we systematically analyze their influence on the selection of instances. Our theoretical and empirical results highlight the crucial role of representations: better representations, e.g., training gradients, generally lead to a better selection of instances, regardless of the chosen selection algorithm. Furthermore, different selection algorithms excel in different settings, and none consistently outperforms the others. Moreover, the selection algorithms do not always align with their intended objectives: for example, algorithms designed for the same objective can select drastically different instances, highlighting the need for careful evaluation.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 4 Jul 2025.

Builds on

37 of 52 references.

References · 37

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  6. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  7. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  8. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  9. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  10. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  11. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  12. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
25 more
  1. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  2. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  3. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  4. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  5. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  6. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  7. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  8. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  9. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  10. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  11. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 183
  12. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  13. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  14. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  15. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  16. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  17. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  18. 2024How to Train Data-Efficient LLMscited by 93
  19. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 44
  20. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  21. 2023Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learningcited by 26
  22. 2023Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasetscited by 19
  23. 2024Dataset Difficulty and the Role of Inductive Biascited by 12
  24. 2023FTFT: Efficient and Robust Fine-Tuning by Transferring Training Dynamicscited by 2
  25. 2023Harnessing Dataset Cartography for Improved Compositional Generalization in Transformerscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Rethinking Representativeness and Diversity in Dynamic Data Selectioncited by 2
  2. 2026Capacity-Dependent Effects of Data Selection for Reasoningcited by 1
  3. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0
  4. 2026Geometric Filtering of LLM-Generated Samples for Few-Shot Text Classificationcited by 0