mlsanityData to 14 Oct 2026

Mosaic-IT: Cost-Free Compositional Data Synthesis for Instruction Tuning

arXiv 2405.13326 ↗First posted 22 May 2024cs.CLcited by 9

On the map: Synthetic data and self-improvement for LLMs → Code and instruction data synthesis

Abstract (arXiv)

Finetuning large language models with a variety of instruction-response pairs has enhanced their capability to understand and follow instructions. Current instruction tuning primarily relies on teacher models or human intervention to generate and refine the instructions and responses for training, which are costly, non-sustainable, and may lack diversity. In this paper, we introduce Mosaic Instruction Tuning (Mosaic-IT), a human/model-free compositional data synthesis method that can efficiently create rich and diverse augmentations from existing instruction tuning data to enhance the LLMs. Mosaic-IT randomly concatenates multiple instruction data into one and trains the model to produce the corresponding responses with predefined higher-level meta-instructions to strengthen its multi-step instruction-following and format-following skills. Our extensive evaluations demonstrate a superior performance and training efficiency of Mosaic-IT, which achieves consistent performance improvements over various benchmarks and an 80% reduction in training costs compared with original instruction tuning. Our codes and data are available at https://github.com/tianyi-lab/Mosaic-IT.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 38 of its 63 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

38 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec

Other references · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
25 more
  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  2. 2023LIMA: Less Is More for Alignmentcited by 1,100
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  9. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  10. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  11. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  12. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  13. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  14. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  15. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  16. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  17. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  18. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 85
  19. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 84
  20. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  21. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  22. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  23. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  24. 2024Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statementscited by 34
  25. 2023Instruction Fusion: Advancing Prompt Evolution through Hybridizationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 1.

9 papers.

  1. 2025Towards Visual Text Grounding of Multimodal Large Language Modelcited by 18
  2. 2024Learning from Committee: Reasoning Distillation from a Mixture of Teachers with Peer-Reviewcited by 14
  3. 2025REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Oncecited by 12
  4. 2025How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradientscited by 8
  5. 2025What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decodingcited by 8
  6. 2025Mitigating Modal Imbalance in Multimodal Reasoningcited by 8
  7. 2025CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoningcited by 4
  8. 2025From Image Generation to Infrastructure Design: a Multi-agent Pipeline for Street Design Generationcited by 3
  9. 2026Synthesizing Instruction-Tuning Datasets with Contrastive Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimizing Instruction Synthesis: Effective Exploration of Evolutionary Space with Tree Search
  2. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
  3. 2025Aligning Instruction Tuning with Pre-training
  4. 2023Instruction Tuning with GPT-4
  5. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning
  6. 2025MAIN: Mutual Alignment Is Necessary for instruction tuning
  7. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  8. 2023Rethinking the Instruction Quality: LIFT is What You Need