# Answering Questions in Stages: Prompt Chaining for Contract QA

arXiv: https://arxiv.org/abs/2410.12840 · HTML: https://mlsanity.com/paper/2410.12840/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 30 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding](https://mlsanity.com/paper/2401.12954/) (2024)
- [Why think step by step? Reasoning emerges from the locality of experience](https://mlsanity.com/paper/2304.03843/) (2023)
- [SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval](https://mlsanity.com/paper/2304.11370/) (2023)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- [Legal Prompt Engineering for Multilingual Legal Judgement Prediction](https://mlsanity.com/paper/2212.02199/) (2022)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [Prompt Chaining or Stepwise Prompt? Refinement in Text Summarization](https://mlsanity.com/paper/2406.00507/) (2024)
- [A Search for Prompts: Generating Structured Answers from Contracts](https://mlsanity.com/paper/2310.10141/) (2023)

## Abstract (arXiv)

Finding answers to legal questions about clauses in contracts is an important form of analysis in many legal workflows (e.g., understanding market trends, due diligence, risk mitigation) but more important is being able to do this at scale. Prior work showed that it is possible to use large language models with simple zero-shot prompts to generate structured answers to questions, which can later be incorporated into legal workflows. Such prompts, while effective on simple and straightforward clauses, fail to perform when the clauses are long and contain information not relevant to the question. In this paper, we propose two-stage prompt chaining to produce structured answers to multiple-choice and multiple-select questions and show that they are more effective than simple prompts on more nuanced legal text. We analyze situations where this technique works well and areas where further refinement is needed, especially when the underlying linguistic variations are more than can be captured by simply specifying possible answers. Finally, we discuss future research that seeks to refine this work by improving stage one results by making them more question-specific.