# Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data Selection

arXiv: https://arxiv.org/abs/2502.11062 · HTML: https://mlsanity.com/paper/2502.11062/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [Training Data Influence Analysis and Estimation: A Survey](https://mlsanity.com/paper/2212.04612/) (2022)
- [#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models](https://mlsanity.com/paper/2308.07074/) (2023)
- [What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions](https://mlsanity.com/paper/2405.13954/) (2024)
- [XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters](https://mlsanity.com/paper/2305.12002/) (2023)
- [Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning](https://mlsanity.com/paper/2407.05040/) (2024)
- [Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning](https://mlsanity.com/paper/2402.11537/) (2024)
- [Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency](https://mlsanity.com/paper/2409.07045/) (2024)
- [Supervised Fine-Tuning Achieve Rapid Task Adaption Via Alternating Attention Head Activation Patterns](https://mlsanity.com/paper/2409.15820/) (2024)

## Built on by (most cited first, 7 of 7)

- [UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection](https://mlsanity.com/paper/2505.12457/) (2025)
- [LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection](https://mlsanity.com/paper/2509.06524/) (2025)
- [Computational Budget Should Be Considered in Data Selection](https://mlsanity.com/paper/2510.16806/) (2025)
- [Form Over Content In Gradient-Based Data Attribution Methods](https://mlsanity.com/paper/2609.19589/) (2026)
- [MADS: Model-Aware Diverse Core Set Selection for Instruction Tuning](https://mlsanity.com/paper/2605.30857/) (2026)
- [How Output Format Confounds Data Quality and Capability in Instruction Tuning](https://mlsanity.com/paper/2609.02015/) (2026)
- [Which Influence Are We Estimating? The Role of Counterfactual Specifications in Data Attribution](https://mlsanity.com/paper/2609.31214/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown great potential across various industries due to their remarkable ability to generalize through instruction tuning. However, the limited availability of domain-specific data significantly hampers their performance on specialized tasks. While existing methods primarily focus on selecting training data from general datasets that are similar to the target domain, they often fail to consider the joint distribution of instructions, resulting in inefficient learning and suboptimal knowledge transfer. To address these challenges, we introduce G2IS (Gradient-based Graph Instruction Selection), a novel method that constructs a mixed gradient-based instruction graph to capture the joint distribution and interdependencies between instructions. By accounting for the relationships between instructions, G2IS improves domain adaptation efficiency. Additionally, we propose a gradient walk algorithm to refine the data selection process, enhancing both training effectiveness and efficiency. Our experiments demonstrate that G2IS outperforms traditional methods across various domain adaptation tasks, yielding significant performance gains, particularly in complex, data-scarce scenarios. These results underscore the potential of G2IS in advancing the development of large, domain-specific models.