mlsanityData to 14 Oct 2026

MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space

arXiv 2504.13835 ↗First posted 18 Apr 2025cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Data quality and diversity are key to the construction of effective instruction-tuning datasets. % With the increasing availability of open-source instruction-tuning datasets, it is advantageous to automatically select high-quality and diverse subsets from a vast amount of data. % Existing methods typically prioritize instance quality and use heuristic rules to maintain diversity. % However, this absence of a comprehensive view of the entire collection often leads to suboptimal results. % Moreover, heuristic rules generally focus on distance or clustering within the embedding space, which fails to accurately capture the intent of complex instructions in the semantic space. % To bridge this gap, we propose a unified method for quantifying the information content of datasets. This method models the semantic space by constructing a label graph and quantifies diversity based on the distribution of information within the graph. % Based on such a measurement, we further introduce an efficient sampling method that selects data samples iteratively to Maximize the Information Gain (MIG) in semantic space. % Experiments on various datasets and base models demonstrate that MIG consistently outperforms state-of-the-art methods. % Notably, the model fine-tuned with 5% Tulu3 data sampled by MIG achieves comparable performance to the official SFT model trained on the full dataset, with improvements of +5.73% on AlpacaEval and +6.89% on Wildbench.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 18 Apr 2025.

Builds on

30 of 46 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  11. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  12. 2023LIMA: Less Is More for Alignmentcited by 1,100
18 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  2. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  3. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  4. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  5. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  6. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  7. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  8. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  9. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  10. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  11. 2024Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimationcited by 53
  12. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  13. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 44
  14. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  15. 2024SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflectioncited by 34
  16. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignmentcited by 33
  17. 2024Diversity Measurement and Subset Selection for Instruction Tuning Datasetscited by 29
  18. 2024Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinementcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2025Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoningcited by 7
  2. 2026SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Trainingcited by 7
  3. 2026Instruction Data Selection via Answer Divergencecited by 3
  4. 2026DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learningcited by 2
  5. 2025GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Datacited by 1
  6. 2026On Representation Redundancy in Large-Scale Instruction Tuning Data Selectioncited by 1
  7. 2026Aligning the Query Space: Greedy Information Projection for Language Model Data Selectioncited by 1
  8. 2026CODEBLOCK: Learning to Supervise Code at the Right Granularitycited by 1
  9. 2026From Tags to Trees: Structuring Fine-Grained Knowledge for Controllable Data Selection in LLM Instruction Tuningcited by 0
  10. 2026Hierarchical Data Selection via Manifold Coverage and Sparse Feature Coverage in LLM Post-trainingcited by 0
  11. 2026Quantifying Error Tolerance in Synthetic Data: An Atomic-level Operand vs. Operator Perturbation Studycited by 0
  12. 2026Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Samplingcited by 0