mlsanityData to 7 Oct 2026

DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection

arXiv 2310.16776 ↗First posted 25 Oct 2023cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent advances have led to the availability of many pre-trained language models (PLMs); however, a question that remains is how much data is truly needed to fine-tune PLMs for downstream tasks? In this work, we introduce DEFT-UCS, a data-efficient fine-tuning framework that leverages unsupervised core-set selection to identify a smaller, representative dataset that reduces the amount of data needed to fine-tune PLMs for downstream tasks. We examine the efficacy of DEFT-UCS in the context of text-editing LMs, and compare to the state-of-the art text-editing model, CoEDIT. Our results demonstrate that DEFT-UCS models are just as accurate as CoEDIT, across eight different datasets consisting of six different editing tasks, while finetuned on 70% less data.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

33 of 46 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
21 more
  1. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  2. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  3. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  4. 2021MetaICL: Learning to Learn In Contextcited by 450
  5. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  6. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  7. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  8. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  9. 2019Automatically Neutralizing Subjective Bias in Textcited by 105
  10. 2020ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformationscited by 101
  11. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 96
  12. 2021RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learningcited by 72
  13. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  14. 2023CoEdIT: Text Editing by Task-Specific Instruction Tuningcited by 67
  15. 2023RewriteLM: An Instruction-Tuned Large Language Model for Text Rewritingcited by 67
  16. 2022Understanding Iterative Revision from Human-Written Textcited by 56
  17. 2019Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Needcited by 43
  18. 2023Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistancecited by 33
  19. 2022EditEval: An Instruction-Based Benchmark for Text Improvementscited by 30
  20. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  21. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Enhancing Training Data Attribution with Representational Optimizationcited by 8
  2. 2025Data-Efficient Hate Speech Detection via Cross-Lingual Nearest Neighbor Retrieval with Limited Labeled Datacited by 2
  3. 2025GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Datacited by 1
  4. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0
  5. 2026MADS: Model-Aware Diverse Core Set Selection for Instruction Tuningcited by 0
  6. 2026STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbationscited by 0