# Learn More, Forget Less: A Gradient-Aware Data Selection Approach for LLM

arXiv: https://arxiv.org/abs/2511.08620 · HTML: https://mlsanity.com/paper/2511.08620/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2025.

## Built on by (most cited first, 5 of 5)

- [Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptation](https://mlsanity.com/paper/2602.06359/) (2026)
- [Reasoning Quality Emerges Early: Data Curation for Reasoning Models](https://mlsanity.com/paper/2606.26797/) (2026)
- [One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning](https://mlsanity.com/paper/2605.06166/) (2026)
- [Two to Tango: Coupled Task-Reference Selection for Safe LLM Fine-tuning](https://mlsanity.com/paper/2606.09866/) (2026)
- [Preference Data Selection for Mitigating the Alignment Tax in Large Language Models](https://mlsanity.com/paper/2608.24192/) (2026)

## Abstract (arXiv)

Despite large language models (LLMs) have achieved impressive achievements across numerous tasks, supervised fine-tuning (SFT) remains essential for adapting these models to specialized domains. However, SFT for domain specialization can be resource-intensive and sometimes leads to a deterioration in performance over general capabilities due to catastrophic forgetting (CF). To address these issues, we propose a self-adaptive gradient-aware data selection approach (GrADS) for supervised fine-tuning of LLMs, which identifies effective subsets of training data by analyzing gradients obtained from a preliminary training phase. Specifically, we design self-guided criteria that leverage the magnitude and statistical distribution of gradients to prioritize examples that contribute the most to the model's learning process. This approach enables the acquisition of representative samples that enhance LLMs understanding of domain-specific tasks. Through extensive experimentation with various LLMs across diverse domains such as medicine, law, and finance, GrADS has demonstrated significant efficiency and cost-effectiveness. Remarkably, utilizing merely 5% of the selected GrADS data, LLMs already surpass the performance of those fine-tuned on the entire dataset, and increasing to 50% of the data results in significant improvements! With catastrophic forgetting substantially mitigated simultaneously. We will release our code for GrADS later.