mlsanityData to 7 Oct 2026

Exploring the Benefits of Training Expert Language Models over Instruction Tuning

arXiv 2302.03202 ↗First posted 7 Feb 2023cs.CLcited by 80

Not on the map. Why?

Abstract (arXiv)

Recently, Language Models (LMs) instruction-tuned on multiple tasks, also known as multitask-prompted fine-tuning (MT), have shown the capability to generalize to unseen tasks. Previous work has shown that scaling the number of training tasks is the key component in making stronger MT LMs. In this work, we report an unexpected finding that an expert LM fine-tuned on just a single task can outperform an MT LM trained with 300+ different tasks on 11 different unseen datasets and on 13 datasets of the BIG-bench benchmark by a mean accuracy of 3.20% and 1.29%, respectively. This finding casts doubt on the previously held belief that simply scaling the number of tasks makes stronger MT LMs. Leveraging this finding, we further show that this distributed approach of training a separate expert LM per training task instead of a single MT LM for zero-shot inference possesses many benefits including (1) avoiding negative task transfer that often occurs during instruction tuning, (2) being able to continually learn new tasks without having to re-train on previous tasks to avoid catastrophic forgetting, and (3) showing compositional capabilities when merging individual experts together. The code is available at https://github.com/joeljang/ELM.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 74 of its 101 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

74 of 101 references.

References · 74

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
38 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  10. 2022Editing Models with Task Arithmeticcited by 1,076
  11. 2021Learning to Prompt for Continual Learningcited by 938
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  13. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  14. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  15. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  16. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  17. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  18. 2019ELI5: Long Form Question Answeringcited by 567
  19. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  20. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  21. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  22. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  23. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  24. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  25. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  26. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  27. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  28. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  29. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  30. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  31. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  32. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  33. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  34. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  35. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  36. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  37. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  38. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 36; 2025: 13; 2026: 5.

80 papers.

Sort:

By area:

  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  2. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  3. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  4. 2023Challenges and Applications of Large Language Modelscited by 251
  5. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  6. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  7. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  8. 2024Continual Learning for Large Language Models: A Surveycited by 144
  9. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  10. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  11. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  12. 2023Deep Model Fusion: A Surveycited by 89
38 more
  1. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  2. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  3. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  4. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  5. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  6. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  7. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  8. 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
  9. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  10. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  11. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  12. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  13. 2023Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuningcited by 56
  14. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  15. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 53
  16. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  17. 2024What Matters for Model Merging at Scale?cited by 52
  18. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  19. 2023Fusing Models with Complementary Expertisecited by 45
  20. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoderscited by 44
  21. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  22. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  23. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  24. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignmentcited by 33
  25. 2023Large Language Model Instruction Following: A Survey of Progresses and Challengescited by 31
  26. 2024Smoothie: Label Free Language Model Routingcited by 31
  27. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  28. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 29
  29. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  30. 2023Instruction Tuned Models are Quick Learnerscited by 21
  31. 2023SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understandingcited by 18
  32. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  33. 2025Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systemscited by 18
  34. 2023OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Trackingcited by 17
  35. 2023Data Management For Training Large Language Models: A Surveycited by 17
  36. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 17
  37. 2024SelectLLM: Can LLMs Select Important Instructions to Annotate?cited by 16
  38. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14

50 most cited of 80 All 80 (JSON)