mlsanityData to 14 Oct 2026

PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents

arXiv 2305.14564 ↗First posted 23 May 2023cs.CLcited by 61

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Strategies such as chain-of-thought prompting improve the performance of large language models (LLMs) on complex reasoning tasks by decomposing input examples into intermediate steps. However, it remains unclear how to apply such methods to reason over long input documents, in which both the decomposition and the output of each intermediate step are non-trivial to obtain. In this work, we propose PEARL, a prompting framework to improve reasoning over long documents, which consists of three stages: action mining, plan formulation, and plan execution. More specifically, given a question about a long document, PEARL decomposes the question into a sequence of actions (e.g., SUMMARIZE, FIND_EVENT, FIND_RELATION) and then executes them over the document to obtain the answer. Each stage of PEARL is implemented via zero-shot or few-shot prompting of LLMs (in our work, GPT-4) with minimal human input. We evaluate PEARL on a challenging subset of the QuALITY dataset, which contains questions that require complex reasoning over long narrative texts. PEARL outperforms zero-shot and chain-of-thought prompting on this dataset, and ablation experiments show that each stage of PEARL is critical to its performance. Overall, PEARL is a first step towards leveraging LLMs to reason over long documents.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 70 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

40 of 70 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models6 May
  2. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks22 Nov
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
23 more
  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  2. 2022Large Language Models Can Self-Improvecited by 664
  3. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  4. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  5. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  6. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  7. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  8. 2023Language Models can Solve Computer Taskscited by 372
  9. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  10. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  11. 2023Large Language Model Guided Tree-of-Thoughtcited by 221
  12. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  13. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  14. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  15. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  16. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  17. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  18. 2022Read before Generate! Faithful Long Form Question Answering with Machine Readingcited by 47
  19. 2021ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answerscited by 29
  20. 2022How Do We Answer Complex Questions: Discourse Structure of Long-form Answerscited by 19
  21. 2021Long Context Question Answering via Supervised Contrastive Learningcited by 17
  22. 2022Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Treescited by 14
  23. 2021Iterative Hierarchical Attention for Answering Complex Questions over Long Documentscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 26; 2025: 17; 2026: 6.

61 papers.

Sort:

By area:

  1. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  2. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 206
  3. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  4. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 177
  5. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 146
  6. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 133
  7. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 83
  8. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  9. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  10. 2024TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generationcited by 51
  11. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  12. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 43
38 more
  1. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaborationcited by 38
  2. 2024Tool-Planner: Task Planning with Clusters across Multiple Toolscited by 33
  3. 2024Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarkscited by 32
  4. 2024CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solvingcited by 31
  5. 2024THREAD: Thinking Deeper with Recursive Spawningcited by 25
  6. 2023Auto-ICL: In-Context Learning without Human Supervisioncited by 18
  7. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 18
  8. 2023A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documentscited by 16
  9. 2025The AI Agent Indexcited by 15
  10. 2025ReCode: Unify Plan and Action for Universal Granularity Controlcited by 15
  11. 2024MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automationcited by 14
  12. 2025RaCoT: Plug-and-Play Contrastive Example Generation Mechanism for Enhanced LLM Reasoning Reliabilitycited by 14
  13. 2024A Modular Approach for Multimodal Summarization of TV Showscited by 13
  14. 2024Learning to Plan for Language Modeling from Unlabeled Datacited by 12
  15. 2024Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videoscited by 12
  16. 2024Mining experimental data from Materials Science literature with Large Language Models: an evaluation studycited by 11
  17. 2024Attribute or Abstain: Large Language Models as Long Document Assistantscited by 11
  18. 2024Can't Remember Details in Long Documents? You Need Some R&Rcited by 10
  19. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Modelscited by 9
  20. 2026SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewardscited by 9
  21. 2023AMERICANO: Argument Generation with Discourse-driven Decomposition and Agent Interactioncited by 8
  22. 2025RedTeamLLM: an Agentic AI framework for offensive securitycited by 7
  23. 2024Will GPT-4 Run DOOM?cited by 6
  24. 2024Sub-goal Distillation: A Method to Improve Small Language Agentscited by 6
  25. 2024Meta Knowledge for Retrieval Augmented Large Language Modelscited by 6
  26. 2025Progressive Document-level Text Simplification via Large Language Modelscited by 5
  27. 2025Select-Then-Decompose: From Empirical Analysis to Adaptive Selection Strategy for Task Decomposition in Large Language Modelscited by 5
  28. 2025Leveraging In-Context Learning for Language Model Agentscited by 4
  29. 2026Agentic Tool Use in Large Language Modelscited by 4
  30. 2024Towards Goal-oriented Prompt Engineering for Large Language Models: A Surveycited by 3
  31. 2025Beyond Pairwise: Global Zero-shot Temporal Graph Generationcited by 3
  32. 2025Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decompositioncited by 3
  33. 2023What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perceptioncited by 2
  34. 2024SEGMENT+: Long Text Processing with Short-Context Language Modelscited by 2
  35. 2023TopicGPT: A Prompt-based Topic Modeling Frameworkcited by 1
  36. 2024Generating Clarification Questions for Disambiguating Contractscited by 1
  37. 2025Graph of Agents: Principled Long Context Modeling by Emergent Multi-Agent Collaborationcited by 1
  38. 2025AuditAgent: Expert-Guided Multi-Agent Reasoning for Cross-Document Fraudulent Evidence Discoverycited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search
  2. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  3. 2022Complexity-Based Prompting for Multi-Step Reasoning
  4. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
  5. 2023Re-Reading Improves Reasoning in Large Language Models
  6. 2025Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
  7. 2024Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Models
  8. 2025Chain-of-Thought Matters: Improving Long-Context Language Models with Reasoning Path Supervision