# TaskLAMA: Probing the Complex Task Understanding of Language Models

arXiv: https://arxiv.org/abs/2308.15299 · HTML: https://mlsanity.com/paper/2308.15299/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 29 Aug 2023.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [Commonsense Knowledge Mining from Pretrained Models](https://mlsanity.com/paper/1909.00505/) (2019)
- [Language Models of Code are Few-Shot Commonsense Learners](https://mlsanity.com/paper/2210.07128/) (2022)
- [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models](https://mlsanity.com/paper/2005.00683/) (2020)
- [Evaluating Commonsense in Pre-trained Language Models](https://mlsanity.com/paper/1911.11931/) (2019)
- [Can Language Models be Biomedical Knowledge Bases?](https://mlsanity.com/paper/2109.07154/) (2021)
- [Dr.ICL: Demonstration-Retrieved In-context Learning](https://mlsanity.com/paper/2305.14128/) (2023)
- [Reasoning about Goals, Steps, and Temporal Ordering with WikiHow](https://mlsanity.com/paper/2009.07690/) (2020)
- [LAMBADA: Backward Chaining for Automated Reasoning in Natural Language](https://mlsanity.com/paper/2212.13894/) (2022)
- [Do Language Embeddings Capture Scales?](https://mlsanity.com/paper/2010.05345/) (2020)
- [BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information](https://mlsanity.com/paper/2306.07934/) (2023)
- [GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models](https://mlsanity.com/paper/2205.12247/) (2022)
- [Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data](https://mlsanity.com/paper/2203.07264/) (2022)
- [proScript: Partially Ordered Scripts Generation via Pre-trained Language Models](https://mlsanity.com/paper/2104.08251/) (2021)
- [SubeventWriter: Iterative Sub-event Sequence Generation with Coherence Controller](https://mlsanity.com/paper/2210.06694/) (2022)
- [Extracting Hierarchies of Search Tasks & Subtasks via a Bayesian Nonparametric Approach](https://mlsanity.com/paper/1706.01574/) (2017)

## Built on by (most cited first, 12 of 12)

- [PlanGenLLMs: A Modern Survey of LLM Planning Capabilities](https://mlsanity.com/paper/2502.11221/) (2025)
- [In-context Learning with Retrieved Demonstrations for Language Models: A Survey](https://mlsanity.com/paper/2401.11624/) (2024)
- [Large Language Models for Planning: A Comprehensive and Systematic Survey](https://mlsanity.com/paper/2505.19683/) (2025)
- [DAG-Plan: Generating Directed Acyclic Dependency Graphs for Dual-Arm Cooperative Planning](https://mlsanity.com/paper/2406.09953/) (2024)
- [Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Services](https://mlsanity.com/paper/2505.18471/) (2025)
- [Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness](https://mlsanity.com/paper/2406.16342/) (2024)
- [PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilities](https://mlsanity.com/paper/2504.14773/) (2025)
- [Open Grounded Planning: Challenges and Benchmark Construction](https://mlsanity.com/paper/2406.02903/) (2024)
- [Scaling LLM Planning: NL2FLOW for Parametric Problem Generation and Rigorous Evaluation](https://mlsanity.com/paper/2507.02253/) (2025)
- [How the Advent of Ubiquitous Large Language Models both Stymie and Turbocharge Dynamic Adversarial Question Generation](https://mlsanity.com/paper/2401.11185/) (2024)
- [Rule Synergy Analysis using LLMs: State of the Art and Implications](https://mlsanity.com/paper/2508.19484/) (2025)
- [Log2Plan: An Adaptive GUI Automation Framework Integrated with Task Mining Approach](https://mlsanity.com/paper/2509.22137/) (2025)

## Abstract (arXiv)

Structured Complex Task Decomposition (SCTD) is the problem of breaking down a complex real-world task (such as planning a wedding) into a directed acyclic graph over individual steps that contribute to achieving the task, with edges specifying temporal dependencies between them. SCTD is an important component of assistive planning tools, and a challenge for commonsense reasoning systems. We probe how accurately SCTD can be done with the knowledge extracted from Large Language Models (LLMs). We introduce a high-quality human-annotated dataset for this problem and novel metrics to fairly assess performance of LLMs against several baselines. Our experiments reveal that LLMs are able to decompose complex tasks into individual steps effectively, with a relative improvement of 15% to 280% over the best baseline. We also propose a number of approaches to further improve their performance, with a relative improvement of 7% to 37% over the base model. However, we find that LLMs still struggle to predict pairwise temporal dependencies, which reveals a gap in their understanding of complex tasks.