mlsanityData to 14 Oct 2026

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

arXiv 2301.13688 ↗First posted 31 Jan 2023cs.AI cs.CL cs.LGcited by 739

Not on the map. Why?

Cited by 70 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study the design decisions of publicly available instruction tuning methods, and break down the development of Flan 2022 (Chung et al., 2022). Through careful ablation studies on the Flan Collection of tasks and methods, we tease apart the effect of design decisions which enable Flan-T5 to outperform prior work by 3-17%+ across evaluation settings. We find task balancing and enrichment techniques are overlooked but critical to effective instruction tuning, and in particular, training with mixed prompt settings (zero-shot, few-shot, and chain-of-thought) actually yields stronger (2%+) performance in all settings. In further experiments, we show Flan-T5 requires less finetuning to converge higher and faster than T5 on single downstream tasks, motivating instruction-tuned models as more computationally-efficient starting checkpoints for new tasks. Finally, to accelerate research on instruction tuning, we make the Flan 2022 collection of datasets, templates, and methods publicly available at https://github.com/google-research/FLAN/tree/main/flan/v2.

Key facts. Cited by 739 papers in the map (data through 14 Oct 2026). Builds on 80 of its 99 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

80 of 99 references.

References · 80

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2018Deep contextualized word representationscited by 2,290
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  13. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  15. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  16. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  17. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  18. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  19. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  20. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  21. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  22. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  23. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  24. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  25. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  26. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  27. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  28. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  29. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  30. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  31. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  32. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  33. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  34. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  35. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  36. 2019Defending Against Neural Fake Newscited by 559
  37. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  38. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 207; 2024: 287; 2025: 165; 2026: 80.

739 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  6. 2023A Survey of Large Language Modelscited by 2,576
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
38 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  6. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  8. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  9. 2023The Falcon Series of Open Language Modelscited by 535
  10. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  11. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  13. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  14. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  15. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  16. 2023Scaling Data-Constrained Language Modelscited by 403
  17. 2023Larger language models do in-context learning differentlycited by 394
  18. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  19. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  20. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  21. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  22. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  23. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  24. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  25. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  26. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  27. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  28. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  29. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  30. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  31. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  32. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  33. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 222
  34. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  35. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  36. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  37. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  38. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197

50 most cited of 739 All 739 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
  2. 2025Aligning Instruction Tuning with Pre-training
  3. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks
  4. 2026Do Instruction-Tuned Models Always Perform Better Than Base Models? Evidence from Math and Domain-Shifted Benchmarks
  5. 2023Rethinking the Instruction Quality: LIFT is What You Need
  6. 2023Dynamics of Instruction Fine-Tuning for Chinese Large Language Models
  7. 2024Optimizing Instruction Synthesis: Effective Exploration of Evolutionary Space with Tree Search
  8. 2023Tuna: Instruction Tuning using Feedback from Large Language Models