mlsanityData to 7 Oct 2026

Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning

arXiv 2306.01150 ↗First posted 1 Jun 2023cs.CL cs.AIcited by 40

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have shown impressive performance in following natural language instructions to solve unseen tasks. However, it remains unclear whether models truly understand task definitions and whether the human-written definitions are optimal. In this paper, we systematically study the role of task definitions in instruction learning. We first conduct an ablation analysis informed by human annotations to understand which parts of a task definition are most important, and find that model performance only drops substantially when removing contents describing the task output, in particular label information. Next, we propose an automatic algorithm to compress task definitions to a minimal supporting set of tokens, and find that 60% of tokens can be removed while maintaining or even improving model performance. Based on these results, we propose two strategies to help models better leverage task instructions: (1) providing only key information for tasks in a common structured format, and (2) adding a meta-tuning stage to help the model better understand the definitions. With these two strategies, we achieve a 4.2 Rouge-L improvement over 119 unseen test tasks.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 13 of its 37 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

13 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?25 Feb

Other references · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2023Instruction Tuning with GPT-4cited by 676
  6. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  7. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  8. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  9. 2023The False Promise of Imitating Proprietary LLMscited by 254
  10. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  11. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  12. 2023Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuningcited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 15; 2025: 8; 2026: 4.

40 papers.

Sort:

By area:

  1. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  2. 2023Explainability for Large Language Models: A Surveycited by 331
  3. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  4. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  5. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 226
  6. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  7. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  8. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  9. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  10. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 53
  11. 2024Symbolic Prompt Program Search: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimizationcited by 29
  12. 2023Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructionscited by 24
28 more
  1. 2025Towards LLMs Robustness to Changes in Prompt Format Stylescited by 21
  2. 2023Data Management For Training Large Language Models: A Surveycited by 17
  3. 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptionscited by 15
  4. 2023Next Steps for Human-Centered Generative AI: A Technical Perspectivecited by 13
  5. 2024How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?cited by 13
  6. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMscited by 13
  7. 2024CliqueParcel: An Approach For Batching LLM Prompts That Jointly Optimizes Efficiency And Faithfulnesscited by 12
  8. 2024Evaluating Human Alignment and Model Faithfulness of LLM Rationalecited by 12
  9. 2023Explore the Potential of LLMs in Misinformation Detection: An Empirical Studycited by 11
  10. 2024Chain-of-Instructions: Compositional Instruction Tuning on Large Language Modelscited by 9
  11. 2024Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Stylescited by 9
  12. 2024Systematic Task Exploration with LLMs: A Study in Citation Text Generationcited by 8
  13. 2025A Fano-Style Accuracy Upper Bound for LLM Single-Pass Reasoning in Multi-Hop QAcited by 8
  14. 2024Improving Event Definition Following For Zero-Shot Event Detectioncited by 7
  15. 2023JsonTuning: Towards Generalizable, Robust, and Controllable Instruction Tuningcited by 6
  16. 2025What Makes a Good Natural Language Prompt?cited by 6
  17. 2024Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMscited by 5
  18. 2024Identifying and Analyzing Performance-Critical Tokens in Large Language Modelscited by 2
  19. 2024How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classescited by 2
  20. 2025Which Words Matter Most in Zero-Shot Prompts?cited by 2
  21. 2025Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelinescited by 2
  22. 2025Facilitating Cognitive Accessibility with LLMs: A Multi-Task Approach to Easy-to-Read Text Generationcited by 2
  23. 2024Deep Exploration of Cross-Lingual Zero-Shot Generalization in Instruction Tuningcited by 1
  24. 2026Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructscited by 1
  25. 2024FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routingcited by 0
  26. 2025Efficient Knowledge Feeding to Language Models: A Novel Integrated Encoder-Decoder Architecturecited by 0
  27. 2026Can Large Language Models Differentiate Harmful from Argumentative Essays? Steps Toward Ethical Essay Scoringcited by 0
  28. 2026Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoning
  2. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment
  3. 2025LLMs can be easily Confused by Instructional Distractions
  4. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
  5. 2026Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding
  6. 2025Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact
  7. 2025Aligning Instruction Tuning with Pre-training
  8. 2025The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives