mlsanityData to 14 Oct 2026

Distilling Script Knowledge from Large Language Models for Constrained Language Planning

arXiv 2305.05252 ↗First posted 9 May 2023cs.CL cs.AIcited by 45

Not on the map. Why?

Abstract (arXiv)

In everyday life, humans often plan their actions by following step-by-step instructions in the form of goal-oriented scripts. Previous work has exploited language models (LMs) to plan for abstract goals of stereotypical activities (e.g., "make a cake"), but leaves more specific goals with multi-facet constraints understudied (e.g., "make a cake for diabetics"). In this paper, we define the task of constrained language planning for the first time. We propose an overgenerate-then-filter approach to improve large language models (LLMs) on this task, and use it to distill a novel constrained language planning dataset, CoScript, which consists of 55,000 scripts. Empirical results demonstrate that our method significantly improves the constrained language planning ability of LLMs, especially on constraint faithfulness. Furthermore, CoScript is demonstrated to be quite effective in endowing smaller LMs with constrained language planning ability.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 36 of its 89 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

36 of 89 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
24 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  3. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  4. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  5. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  6. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  7. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  8. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  9. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 138
  10. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  11. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  12. 2020Content Planning for Neural Story Generation with Aristotelian Rescoringcited by 92
  13. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  14. 2022Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning taskscited by 56
  15. 2021Your fairness may vary: Pretrained language model fairness in toxic text classificationcited by 39
  16. 2020A Dataset for Tracking Entities in Open Domain Procedural Textcited by 37
  17. 2021Goal-Oriented Script Constructioncited by 30
  18. 2022PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generationcited by 30
  19. 2022A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generationcited by 28
  20. 2021GPT3-to-plan: Extracting plans from text using GPT-3cited by 26
  21. 2019Prospection: Interpretable Plans From Language By Predicting the Futurecited by 21
  22. 2021proScript: Partially Ordered Scripts Generation via Pre-trained Language Modelscited by 21
  23. 2021What do Large Language Models Learn about Scripts?cited by 19
  24. 2021Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrievalcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 20; 2025: 13; 2026: 2.

45 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  3. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  4. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  5. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  6. 2024EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithmscited by 113
  7. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 83
  8. 2023Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arenacited by 78
  9. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  10. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 49
  11. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 47
  12. 2024MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Datacited by 43
33 more
  1. 2024MIRAI: Evaluating LLM Agents for Event Forecastingcited by 41
  2. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  3. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  4. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 39
  5. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  6. 2023Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Modelscited by 24
  7. 2024TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulationcited by 23
  8. 2024IDEA: Enhancing the Rule Learning Ability of Large Language Model Agent through Induction, Deduction, and Abductioncited by 22
  9. 2024SelfGoal: Your Language Agents Already Know How to Achieve High-level Goalscited by 21
  10. 2025LLM-Powered Decentralized Generative Agents with Adaptive Hierarchical Knowledge Graph for Cooperative Planningcited by 18
  11. 2024MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Datasetcited by 11
  12. 2025EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Associationcited by 10
  13. 2024LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planningcited by 9
  14. 2025PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimationcited by 8
  15. 2025TCP: a Benchmark for Temporal Constraint-Based Planningcited by 8
  16. 2025PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilitiescited by 7
  17. 2025Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Predictioncited by 6
  18. 2023Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customizationcited by 5
  19. 2024ERABAL: Enhancing Role-Playing Agents through Boundary-Aware Learningcited by 5
  20. 2025A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysiscited by 5
  21. 2023PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planningcited by 4
  22. 2025Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsitycited by 4
  23. 2023Do large language models and humans have similar behaviors in causal inference with script knowledge?cited by 3
  24. 2025Do LLMs Align Human Values Regarding Social Biases? Judging and Explaining Social Biases with LLMscited by 3
  25. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  26. 2024Diverse and Fine-Grained Instruction-Following Ability Exploration with Synthetic Datacited by 2
  27. 2024REVECA: Adaptive Planning and Trajectory-based Validation in Cooperative Language Agents using Information Relevance and Relative Proximitycited by 1
  28. 2024Past Meets Present: Creating Historical Analogy with Large Language Modelscited by 1
  29. 2025Sample-aware Adaptive Structured Pruning for Large Language Modelscited by 1
  30. 2025ScEdit: Script-based Assessment of Knowledge Editingcited by 1
  31. 2024Persona Knowledge-Aligned Prompt Tuning Method for Online Debatecited by 0
  32. 2026SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluationcited by 0
  33. 2026ADE: Agentic Data Evolution Framework for Human-Centered Objectivescited by 0