mlsanityData to 7 Oct 2026

Large Language Model Instruction Following: A Survey of Progresses and Challenges

arXiv 2303.10475 ↗First posted 18 Mar 2023cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

Task semantics can be expressed by a set of input-output examples or a piece of textual instruction. Conventional machine learning approaches for natural language processing (NLP) mainly rely on the availability of large-scale sets of task-specific examples. Two issues arise: first, collecting task-specific labeled examples does not apply to scenarios where tasks may be too complicated or costly to annotate, or the system is required to handle a new task immediately; second, this is not user-friendly since end-users are probably more willing to provide task description rather than a set of examples before using the system. Therefore, the community is paying increasing interest in a new supervision-seeking paradigm for NLP: learning to follow task instructions, i.e., instruction following. Despite its impressive progress, there are some common issues that the community struggles with. This survey paper tries to summarize and provide insights to the current research on instruction following, particularly, by answering the following questions: (i) What is task instruction, and what instruction types exist? (ii) How to model instructions? (iii) What are popular instruction following datasets and evaluation metrics? (iv) What factors influence and explain the instructions' performance? (v) What challenges remain in instruction following? To our knowledge, this is the first comprehensive survey about instruction following.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 111 of its 212 references that are in the map. First posted on arXiv 18 Mar 2023.

Builds on

111 of 212 references.

References · 111

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  13. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  14. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  15. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  16. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  17. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  18. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  19. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  20. 2016HyperNetworkscited by 1,003
  21. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  22. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  23. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  24. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  25. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  26. 2022A Survey on In-context Learningcited by 717
  27. 2021GPT Understands, Toocited by 715
  28. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  29. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  30. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  31. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  32. 2021MetaICL: Learning to Learn In Contextcited by 450
  33. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  34. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  35. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  36. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  37. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  38. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324

50 most cited of 111 All 111 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 6; 2025: 11; 2026: 8.

31 papers.

Sort:

By area:

  1. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  2. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  3. 2023Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Modelscited by 67
  4. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  5. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  6. 2024SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Modelscited by 23
  7. 2024Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contextscited by 19
  8. 2025How Many Instructions Can LLMs Follow at Once?cited by 17
  9. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  10. 2026SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinementcited by 11
  11. 2025Language Models Can Predict Their Own Behaviorcited by 8
  12. 2025Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Followingcited by 6
19 more
  1. 2025Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systemscited by 6
  2. 2023Learning to Rank Context for Named Entity Recognition Using a Synthetic Datasetcited by 5
  3. 2025Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generationcited by 5
  4. 2025MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agentscited by 3
  5. 2026Structure Enables Effective Self-Localization of Errors in LLMscited by 3
  6. 2024Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context Learningcited by 2
  7. 2024In-Context Learning for Preserving Patient Privacy: A Framework for Synthesizing Realistic Patient Portal Messagescited by 2
  8. 2025LLMs that Understand Processes: Instruction-tuning for Semantics-Aware Process Miningcited by 2
  9. 2023Evaluating the Robustness to Instructions of Large Language Modelscited by 1
  10. 2025Follow My Lead: Logical Fallacy Classification with Knowledge-Augmented LLMscited by 1
  11. 2025Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixturescited by 1
  12. 2025Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Taskscited by 0
  13. 2025Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMscited by 0
  14. 2026LEAF-SQL: Level-wise Exploration with Adaptive Fine-graining for Text-to-SQL Skeleton Predictioncited by 0
  15. 2026MiCU: End-to-End Smart Home Command Understanding with Large Language Modelcited by 0
  16. 2026SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Informationcited by 0
  17. 2026Auditing Privacy Risks in LLM-Enhanced Graph Neural Networkscited by 0
  18. 2026Cross-Relational Preference Learning for Better LLM Instruction Followingcited by 0
  19. 2026ScopeIF: Improving Scope-Aware Precise Instruction-Following in Large Language Models via Graded Reward Modelingcited by 0