# PatentEval: Understanding Errors in Patent Generation

arXiv: https://arxiv.org/abs/2406.06589 · HTML: https://mlsanity.com/paper/2406.06589/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization](https://mlsanity.com/paper/1906.03741/) (2019)
- [Patent Claim Generation by Fine-Tuning OpenAI GPT-2](https://mlsanity.com/paper/1907.02052/) (2019)
- [FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations](https://mlsanity.com/paper/2204.06508/) (2022)
- [PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model](https://mlsanity.com/paper/1906.02124/) (2019)
- [PatentMatch: A Dataset for Matching Patent Claims & Prior Art](https://mlsanity.com/paper/2012.13919/) (2020)
- [Prior Art Search and Reranking for Generated Patent Text](https://mlsanity.com/paper/2009.09132/) (2020)
- [Summarization, Simplification, and Generation: The Case of Patents](https://mlsanity.com/paper/2104.14860/) (2021)
- [PatentTransformer-2: Controlling Patent Text Generation by Structural Metadata](https://mlsanity.com/paper/2001.03708/) (2020)
- [Measuring Patent Claim Generation by Span Relevancy](https://mlsanity.com/paper/1908.09591/) (2019)

## Built on by (most cited first, 3 of 3)

- [AutoSpec: An Agentic Framework for Automatically Drafting Patent Specification](https://mlsanity.com/paper/2509.19640/) (2025)
- [Benchmarking Patent Drafting from Inventor-Style Disclosures](https://mlsanity.com/paper/2608.21249/) (2026)
- [LogicTree-RAG: Logic Tree-guided Retrieval-Augmented Generation for Long-form Patent Drafting](https://mlsanity.com/paper/2609.30943/) (2026)

## Abstract (arXiv)

In this work, we introduce a comprehensive error typology specifically designed for evaluating two distinct tasks in machine-generated patent texts: claims-to-abstract generation, and the generation of the next claim given previous ones. We have also developed a benchmark, PatentEval, for systematically assessing language models in this context. Our study includes a comparative analysis, annotated by humans, of various models. These range from those specifically adapted during training for tasks within the patent domain to the latest general-purpose large language models (LLMs). Furthermore, we explored and evaluated some metrics to approximate human judgments in patent text evaluation, analyzing the extent to which these metrics align with expert assessments. These approaches provide valuable insights into the capabilities and limitations of current language models in the specialized field of patent text generation.