mlsanityData to 14 Oct 2026

Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learning

arXiv 2505.12212 ↗First posted 18 May 2025cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models (LLMs) on task-specific data is essential for their effective deployment. As dataset sizes grow, efficiently selecting optimal subsets for training becomes crucial to balancing performance and computational costs. Traditional data selection methods often require fine-tuning a scoring model on the target dataset, which is time-consuming and resource-intensive, or rely on heuristics that fail to fully leverage the model's predictive capabilities. To address these challenges, we propose Data Whisperer, an efficient, training-free, attention-based method that leverages few-shot in-context learning with the model to be fine-tuned. Comprehensive evaluations were conducted on both raw and synthetic datasets across diverse tasks and models. Notably, Data Whisperer achieves superior performance compared to the full GSM8K dataset on the Llama-3-8B-Instruct model, using just 10% of the data, and outperforms existing methods with a 3.1-point improvement and a 7.4× speedup. The code is available at https://github.com/gszfwsb/Data-Whisperer.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

40 of 56 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  12. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
28 more
  1. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  2. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  3. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  4. 2020Dataset Condensation with Gradient Matchingcited by 516
  5. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  7. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  8. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  9. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  10. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  11. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  12. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  13. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 183
  14. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  15. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  16. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
  17. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  18. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  19. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  20. 2023Data Pruning via Moving-one-Sample-outcited by 67
  21. 2025Dataset Distillation with Neural Characteristic Function: A Minmax Perspectivecited by 45
  22. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 44
  23. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 22
  24. 2024A CLIP-Powered Framework for Robust and Generalizable Data Selectioncited by 21
  25. 2023In-context Learning and Gradient Descent Revisitedcited by 20
  26. 2024Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillationcited by 14
  27. 2024Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinementcited by 13
  28. 2024Speculative Coreset Selection for Task-Specific Fine-tuningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 9.

20 papers.

Sort:

By area:

  1. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 93
  2. 2025Efficient Multi-modal Large Language Models via Progressive Consistency Distillationcited by 22
  3. 2025Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuningcited by 8
  4. 2025Distributional Machine Unlearning via Selective Data Removalcited by 5
  5. 2025Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selectioncited by 4
  6. 2025ImagebindDC: Compressing Multi-modal Data with Imagebind-based Condensationcited by 4
  7. 2025ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuningcited by 3
  8. 2025Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMscited by 2
  9. 2025OmniDocLayout: Towards Diverse Document Layout Generation via Coarse-to-Fine LLM Learningcited by 2
  10. 2025Logical Structure as Knowledge: Enhancing LLM Reasoning via Structured Logical Knowledge Density Estimationcited by 1
  11. 2026GRACE: A Dynamic Coreset Selection Framework for Large Language Model Optimizationcited by 1
  12. 2025UNSEEN: Enhancing Dataset Pruning from a Generalization Perspectivecited by 0
8 more
  1. 2026Is One-Shot In-Context Learning Helpful for Data Selection in Task-Specific Fine-Tuning of Multimodal LLMs?cited by 0
  2. 2026Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxiescited by 0
  3. 2026GRIP-VLM: Group-Relative Importance Pruning for Efficient Vision-Language Modelscited by 0
  4. 2026From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignmentcited by 0
  5. 2026Cross-Domain Feature Expansion for Tabular Medical Data via Knowledge Graphs Injectioncited by 0
  6. 2026PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoningcited by 0
  7. 2026Neuron-Guided Fine-Tuning: Unlocking Efficient Alignment Mechanisms for Large Language Modelscited by 0
  8. 2026Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Modelscited by 0