mlsanityData to 14 Oct 2026

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain

arXiv 2505.14826 ↗First posted 20 May 2025cs.LG cs.CL stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

Supervised fine-tuning (SFT) is a standard approach to adapting large language models (LLMs) to new domains. In this work, we improve the statistical efficiency of SFT by selecting an informative subset of training examples. Specifically, for a fixed budget of training examples, which determines the computational cost of fine-tuning, we determine the most informative ones. The key idea in our method is to select examples that maximize information gain, measured by the Hessian of the log-likelihood of the LLM. We approximate it efficiently by linearizing the LLM at the last layer using multinomial logistic regression models. Our approach is computationally efficient, analyzable, and performs well empirically. We demonstrate this on several problems, and back our claims with both quantitative results and an LLM evaluation.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

31 of 49 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  8. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  9. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  10. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  11. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  12. 2023Scaling Data-Constrained Language Modelscited by 403
19 more
  1. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  2. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  4. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  5. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 192
  6. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  7. 2024How to Train Data-Efficient LLMscited by 93
  8. 2016Coresets and Sketchescited by 63
  9. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  10. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  11. 2019Discrepancy, Coresets, and Sketches in Machine Learningcited by 30
  12. 2024Optimal Design for Reward Modeling in RLHFcited by 25
  13. 2014Minimax-optimal Inference from Partial Rankingscited by 24
  14. 2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 20
  15. 2024Optimal Design for Human Preference Elicitationcited by 17
  16. 2024Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyondcited by 15
  17. 2019Sub-linear RACE Sketches for Approximate Kernel Density Estimation on Streaming Datacited by 14
  18. 2021SVP-CF: Selection via Proxy for Collaborative Filtering Datacited by 14
  19. 2024Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 15
  2. 2025How Reliable is Language Model Micro-Benchmarking?cited by 9
  3. 2026SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Trainingcited by 7
  4. 2026Learning from Mistakes: Negative Reasoning Samples Enhance Out-of-Domain Generalizationcited by 5
  5. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  6. 2025RL-Guided Data Selection for Language Model Finetuningcited by 2
  7. 2025Epistemic diversity across language models mitigates knowledge collapsecited by 2
  8. 2026Aligning the Query Space: Greedy Information Projection for Language Model Data Selectioncited by 1
  9. 2025Spectral Identifiability for Interpretable Probe Geometrycited by 0
  10. 2026Mashup Learning: Faster Finetuning by Remixing Past Checkpointscited by 0
  11. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0
  12. 2026Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Trainingcited by 0
3 more
  1. 2026Hierarchical Data Selection via Manifold Coverage and Sparse Feature Coverage in LLM Post-trainingcited by 0
  2. 2026Online Self-Weighted Fine-Tuningcited by 0
  3. 2026Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Modelscited by 0