mlsanityData to 7 Oct 2026

Instructional Text Across Disciplines: A Survey of Representations, Downstream Tasks, and Open Challenges Toward Capable AI Agents

arXiv 2410.18529 ↗First posted 24 Oct 2024cs.CLcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Recent advances in large language models have demonstrated promising capabilities in following simple instructions through instruction tuning. However, real-world tasks often involve complex, multi-step instructions that remain challenging for current NLP systems. Robust understanding of such instructions is essential for deploying LLMs as general-purpose agents that can be programmed in natural language to perform complex, real-world tasks across domains like robotics, business automation, and interactive systems. Despite growing interest in this area, there is a lack of a comprehensive survey that systematically analyzes the landscape of complex instruction understanding and processing. Through a systematic review of the literature, we analyze available resources, representation schemes, and downstream tasks related to instructional text. Our study examines 181 papers, identifying trends, challenges, and opportunities in this emerging field. We provide AI/NLP researchers with essential background knowledge and a unified view of various approaches to complex instruction understanding, bridging gaps between different research directions and highlighting future research opportunities.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 43 of its 263 references that are in the map. First posted on arXiv 24 Oct 2024.

Builds on

43 of 263 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents5 Mar

Other references · 42

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  6. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  7. 2024A Survey on Large Language Models for Code Generationcited by 646
  8. 2023A Comprehensive Overview of Large Language Modelscited by 639
  9. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  10. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  11. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  12. 2023Language Models can Solve Computer Taskscited by 371
30 more
  1. 2017Embodied Question Answeringcited by 335
  2. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  3. 2024Latent Action Pretraining from Videoscited by 302
  4. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  5. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  6. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  7. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  8. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  9. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  10. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  11. 2018A Survey on Semantic Parsingcited by 62
  12. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  13. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  14. 2019Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Modelscited by 36
  15. 2023Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interactioncited by 35
  16. 2019Hierarchical Decision Making by Generating and Following Natural Language Instructionscited by 30
  17. 2024Process Modeling With Large Language Modelscited by 26
  18. 2024A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddingscited by 23
  19. 2024Associative Recurrent Memory Transformercited by 15
  20. 2023Lexi: Self-Supervised Learning of the UI Languagecited by 10
  21. 2024CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Planscited by 10
  22. 2021HowSumm: A Multi-Document Summarization Dataset Derived from WikiHow Articlescited by 8
  23. 2022Reasoning about Procedures with Natural Language Processing: A Tutorialcited by 8
  24. 2018What Happened? Leveraging VerbNet to Predict the Effects of Actions in Procedural Textcited by 5
  25. 2024WorldAPIs: The World Is Worth How Many APIs? A Thought Experimentcited by 4
  26. 2022Summarizing a virtual robot's past actions in natural languagecited by 3
  27. 2024Learning to Recover from Plan Execution Errors during Robot Manipulation: A Neuro-symbolic Approachcited by 3
  28. 2019A context-aware knowledge acquisition for planning applications using ontologiescited by 2
  29. 2021Tracking entities in technical procedures -- a new dataset and baselinescited by 2
  30. 2023Multi-Step Dialogue Workflow Action Predictioncited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Generating Structured Plan Representation of Procedures with LLMscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024Do LLMs "know" internally when they follow instructions?
  2. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs
  3. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios
  4. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment
  5. 2026Enhancing LLM Instruction Following: An Evaluation-Driven Multi-Agentic Workflow for Prompt Instructions Optimization
  6. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users
  7. 2025Simulating Students with Large Language Models: A Review of Architecture, Mechanisms, and Role Modelling in Education with Generative AI
  8. 2026SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge