mlsanityData to 14 Oct 2026

Demystifying Instruction Mixing for Fine-tuning Large Language Models

arXiv 2312.10793 ↗First posted 17 Dec 2023cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Instruction tuning significantly enhances the performance of large language models (LLMs) across various tasks. However, the procedure to optimizing the mixing of instruction datasets for LLM fine-tuning is still poorly understood. This study categorizes instructions into three primary types: NLP downstream tasks, coding, and general chat. We explore the effects of instruction tuning on different combinations of datasets on LLM performance, and find that certain instruction types are more advantageous for specific applications but can negatively impact other areas. This work provides insights into instruction mixtures, laying the foundations for future research.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 17 Dec 2023.

Builds on

17 of 31 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  10. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
5 more
  1. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  2. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  3. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  4. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  5. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  2. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 10
  3. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Modelscited by 5
  4. 2025Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalitiescited by 5