# Large Language Model Prompt Chaining for Long Legal Document Classification

arXiv: https://arxiv.org/abs/2308.04138 · HTML: https://mlsanity.com/paper/2308.04138/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 8 Aug 2023.

## Builds on (most cited first, 29 of 29)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [OpenPrompt: An Open-source Framework for Prompt-learning](https://mlsanity.com/paper/2111.01998/) (2021)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence](https://mlsanity.com/paper/2004.12158/) (2020)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [GPT Takes the Bar Exam](https://mlsanity.com/paper/2212.14402/) (2022)
- [Legal Prompting: Teaching a Language Model to Think Like a Lawyer](https://mlsanity.com/paper/2212.01326/) (2022)
- [Legal Prompt Engineering for Multilingual Legal Judgement Prediction](https://mlsanity.com/paper/2212.02199/) (2022)
- [Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities](https://mlsanity.com/paper/2206.10883/) (2022)
- [An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification](https://mlsanity.com/paper/2210.05529/) (2022)
- [ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark](https://mlsanity.com/paper/2304.12202/) (2023)
- [Efficient Classification of Long Documents Using Transformers](https://mlsanity.com/paper/2203.11258/) (2022)
- [Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformer](https://mlsanity.com/paper/2211.00974/) (2022)
- [Comparative Study of Long Document Classification](https://mlsanity.com/paper/2111.00702/) (2021)

## Built on by (most cited first, 7 of 7)

- [Interpretable Cross-Examination Technique (ICE-T): Using highly informative features to boost LLM performance](https://mlsanity.com/paper/2405.06703/) (2024)
- [LegalGuardian: A Privacy-Preserving Framework for Secure Integration of Large Language Models in Legal Practice](https://mlsanity.com/paper/2501.10915/) (2025)
- [Opportunities and Challenges of Applying Large Language Models in Building Energy Efficiency and Decarbonization Studies: An Exploratory Overview](https://mlsanity.com/paper/2312.11701/) (2023)
- [Measuring the Groundedness of Legal Question-Answering Systems](https://mlsanity.com/paper/2410.08764/) (2024)
- [Conditional Multi-Stage Failure Recovery for Embodied Agents](https://mlsanity.com/paper/2507.06016/) (2025)
- [Towards Supporting Legal Argumentation with NLP: Is More Data Really All You Need?](https://mlsanity.com/paper/2406.10974/) (2024)
- [Legal Document Summarization: Enhancing Judicial Efficiency through Automation Detection](https://mlsanity.com/paper/2507.18952/) (2025)

## Abstract (arXiv)

Prompting is used to guide or steer a language model in generating an appropriate response that is consistent with the desired outcome. Chaining is a strategy used to decompose complex tasks into smaller, manageable components. In this study, we utilize prompt chaining for extensive legal document classification tasks, which present difficulties due to their intricate domain-specific language and considerable length. Our approach begins with the creation of a concise summary of the original document, followed by a semantic search for related exemplar texts and their corresponding annotations from a training corpus. Finally, we prompt for a label - based on the task - to assign, by leveraging the in-context learning from the few-shot prompt. We demonstrate that through prompt chaining, we can not only enhance the performance over zero-shot, but also surpass the micro-F1 score achieved by larger models, such as ChatGPT zero-shot, using smaller models.