A Search for Prompts: Generating Structured Answers from Contracts
Not on the map. Why?
Abstract (arXiv)
In many legal processes being able to action on the concrete implication of a legal question can be valuable to automating human review or signalling certain conditions (e.g., alerts around automatic renewal). To support such tasks, we present a form of legal question answering that seeks to return one (or more) fixed answers for a question about a contract clause. After showing that unstructured generative question answering can have questionable outcomes for such a task, we discuss our exploration methodology for legal question answering prompts using OpenAI's textitGPT-3.5-Turbo and provide a summary of insights. Using insights gleaned from our qualitative experiences, we compare our proposed template prompts against a common semantic matching approach and find that our prompt templates are far more accurate despite being less reliable in the exact response return. With some additional tweaks to prompts and the use of in-context learning, we are able to further improve the performance of our proposed strategy while maximizing the reliability of responses as best we can.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 16 Oct 2023.
Builds on
21 of 38 references.
References · 21
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
- 2023PaLM 2 Technical Reportcited by 1,162
9 more
- 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
- 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
- 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
- 2023Benchmarking Large Language Models for News Summarizationcited by 445
- 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
- 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 308
- 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
- 2022Legal Prompting: Teaching a Language Model to Think Like a Lawyercited by 60
Built on by
Citing papers per year: 2024: 1.
1 paper.