# Patent Claim Generation by Fine-Tuning OpenAI GPT-2

arXiv: https://arxiv.org/abs/1907.02052 · HTML: https://mlsanity.com/paper/1907.02052/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2019.

## Built on by (most cited first, 30 of 49)

- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Do Language Models Plagiarize?](https://mlsanity.com/paper/2203.07618/) (2022)
- [CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequences](https://mlsanity.com/paper/2202.06689/) (2022)
- [Generating Fake Cyber Threat Intelligence Using Transformer-Based Models](https://mlsanity.com/paper/2102.04351/) (2021)
- [Generative Transformers for Design Concept Generation](https://mlsanity.com/paper/2211.03468/) (2022)
- [Natural Language Processing in the Patent Domain: A Survey](https://mlsanity.com/paper/2403.04105/) (2024)
- [InstructPatentGPT: Training patent language models to follow instructions with human feedback](https://mlsanity.com/paper/2406.16897/) (2024)
- [Can Large Language Models Generate High-quality Patent Claims?](https://mlsanity.com/paper/2406.19465/) (2024)
- [Generative Pre-Trained Transformer for Design Concept Generation: An Exploration](https://mlsanity.com/paper/2111.08489/) (2021)
- [FETA: Towards Specializing Foundation Models for Expert Task Applications](https://mlsanity.com/paper/2209.03648/) (2022)
- [PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims](https://mlsanity.com/paper/2505.19345/) (2025)
- [A Survey on Patent Analysis: From NLP to Multimodal AI](https://mlsanity.com/paper/2404.08668/) (2024)
- [LexGPT 0.1: pre-trained GPT-J models with Pile of Law](https://mlsanity.com/paper/2306.05431/) (2023)
- [AE-GPT: Using Large Language Models to Extract Adverse Events from Surveillance Reports-A Use Case with Influenza Vaccine Adverse Events](https://mlsanity.com/paper/2309.16150/) (2023)
- [Towards Better Evaluation for Generated Patent Claims](https://mlsanity.com/paper/2505.11095/) (2025)
- [Summarization, Simplification, and Generation: The Case of Patents](https://mlsanity.com/paper/2104.14860/) (2021)
- [Evaluating Generative Patent Language Models](https://mlsanity.com/paper/2206.14578/) (2022)
- [Event-based Dynamic Graph Representation Learning for Patent Application Trend Prediction](https://mlsanity.com/paper/2308.09780/) (2023)
- [Patent Data for Engineering Design: A Critical Review and Future Directions](https://mlsanity.com/paper/2111.08500/) (2021)
- [Enriching Patent Claim Generation with European Patent Dataset](https://mlsanity.com/paper/2505.12568/) (2025)
- [Punctuation and Predicates in Language Models](https://mlsanity.com/paper/2508.14067/) (2025)
- [Automatic Claim Review for Climate Science via Explanation Generation](https://mlsanity.com/paper/2107.14740/) (2021)
- [Large Language Model for Patent Concept Generation](https://mlsanity.com/paper/2409.00092/) (2024)
- [Patent-CR: A Dataset for Patent Claim Revision](https://mlsanity.com/paper/2412.02549/) (2024)
- [From PARIS to LE-PARIS: Toward Patent Response Automation with Recommender Systems and Collaborative Large Language Models](https://mlsanity.com/paper/2402.00421/) (2024)
- [Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph](https://mlsanity.com/paper/2404.14372/) (2024)
- [PatentEval: Understanding Errors in Patent Generation](https://mlsanity.com/paper/2406.06589/) (2024)
- [AutoSpec: An Agentic Framework for Automatically Drafting Patent Specification](https://mlsanity.com/paper/2509.19640/) (2025)
- [Pat-DEVAL: Chain-of-Legal-Thought Evaluation for Patent Description](https://mlsanity.com/paper/2601.00166/) (2026)
- [Hybrid approach to detecting symptoms of depression in social media entries](https://mlsanity.com/paper/2106.10485/) (2021)

## Abstract (arXiv)

In this work, we focus on fine-tuning an OpenAI GPT-2 pre-trained model for generating patent claims. GPT-2 has demonstrated impressive efficacy of pre-trained language models on various tasks, particularly coherent text generation. Patent claim language itself has rarely been explored in the past and poses a unique challenge. We are motivated to generate coherent patent claims automatically so that augmented inventing might be viable someday. In our implementation, we identified a unique language structure in patent claims and leveraged its implicit human annotations. We investigated the fine-tuning process by probing the first 100 steps and observing the generated text at each step. Based on both conditional and unconditional random sampling, we analyze the overall quality of generated patent claims. Our contributions include: (1) being the first to generate patent claims by machines and being the first to apply GPT-2 to patent claim generation, (2) providing various experiment results for qualitative analysis and future research, (3) proposing a new sampling approach for text generation, and (4) building an e-mail bot for future researchers to explore the fine-tuned GPT-2 model further.