# The Harvard USPTO Patent Dataset: A Large-Scale, Well-Structured, and Multi-Purpose Corpus of Patent Applications

arXiv: https://arxiv.org/abs/2207.04043 · HTML: https://mlsanity.com/paper/2207.04043/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 64 references that are in the map.
- First posted on arXiv 8 Jul 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Captum: A unified and generic model interpretability library for PyTorch](https://mlsanity.com/paper/2009.07896/) (2020)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization](https://mlsanity.com/paper/1906.03741/) (2019)
- [Controlling Text Edition by Changing Answers of Specific Questions](https://mlsanity.com/paper/2105.11018/) (2021)

## Built on by (most cited first, 30 of 33)

- [Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2211.07634/) (2022)
- [Natural Language Processing in the Patent Domain: A Survey](https://mlsanity.com/paper/2403.04105/) (2024)
- [Can Large Language Models Generate High-quality Patent Claims?](https://mlsanity.com/paper/2406.19465/) (2024)
- [DrugPilot: LLM-based Parameterized Reasoning Agent for Drug Discovery](https://mlsanity.com/paper/2505.13940/) (2025)
- [BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts](https://mlsanity.com/paper/2408.08274/) (2024)
- [A Comparative Study of Quality Evaluation Methods for Text Summarization](https://mlsanity.com/paper/2407.00747/) (2024)
- [PatentScore: Multi-dimensional Evaluation of LLM-Generated Patent Claims](https://mlsanity.com/paper/2505.19345/) (2025)
- [A Survey on Patent Analysis: From NLP to Multimodal AI](https://mlsanity.com/paper/2404.08668/) (2024)
- [string2string: A Modern Python Library for String-to-String Algorithms](https://mlsanity.com/paper/2304.14395/) (2023)
- [Towards Better Evaluation for Generated Patent Claims](https://mlsanity.com/paper/2505.11095/) (2025)
- [PatentLMM: Large Multimodal Model for Generating Descriptions for Patent Figures](https://mlsanity.com/paper/2501.15074/) (2025)
- [HDT: Hierarchical Document Transformer](https://mlsanity.com/paper/2407.08330/) (2024)
- [Multi-Scale Heterogeneous Text-Attributed Graph Datasets From Diverse Domains](https://mlsanity.com/paper/2412.08937/) (2024)
- [Enriching Patent Claim Generation with European Patent Dataset](https://mlsanity.com/paper/2505.12568/) (2025)
- [PILOT-Bench: A Benchmark for Legal Reasoning in the Patent Domain with IRAC-Aligned Classification Tasks](https://mlsanity.com/paper/2601.04758/) (2026)
- [IPEval: A Bilingual Intellectual Property Agency Consultation Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2406.12386/) (2024)
- [Towards Automated Patent Workflows: AI-Orchestrated Multi-Agent Framework for Intellectual Property Management and Analysis](https://mlsanity.com/paper/2409.19006/) (2024)
- [PatentEdits: Framing Patent Novelty as Textual Entailment](https://mlsanity.com/paper/2411.13477/) (2024)
- [Can AI Examine Novelty of Patents?: Novelty Evaluation Based on the Correspondence between Patent Claim and Prior Art](https://mlsanity.com/paper/2502.06316/) (2025)
- [AutoSpec: An Agentic Framework for Automatically Drafting Patent Specification](https://mlsanity.com/paper/2509.19640/) (2025)
- [Patent Language Model Pretraining with ModernBERT](https://mlsanity.com/paper/2509.14926/) (2025)
- [Monolingual and Cross-Lingual Knowledge Transfer for Topic Classification](https://mlsanity.com/paper/2306.07797/) (2023)
- [AIMS.au: A Dataset for the Analysis of Modern Slavery Countermeasures in Corporate Statements](https://mlsanity.com/paper/2502.07022/) (2025)
- [Adaptive Cost-Efficient Evaluation for Reliable Patent Claim Generation](https://mlsanity.com/paper/2604.04295/) (2026)
- [Pointer-Augmented Autoregressive Generation of Patent Claims with Joint Topology and Content Decoding](https://mlsanity.com/paper/2607.24040/) (2026)
- [Extracting Text Representations for Terms and Phrases in Technical Domains](https://mlsanity.com/paper/2305.15867/) (2023)
- [DiTTO-LLM: Framework for Discovering Topic-based Technology Opportunities via Large Language Model](https://mlsanity.com/paper/2509.09724/) (2025)
- [It's LIT! Reliability-Optimized LLMs with Inspectable Tools](https://mlsanity.com/paper/2511.14903/) (2025)
- [Adaptive Multi-Stage Patent Claim Generation with Unified Quality Assessment](https://mlsanity.com/paper/2601.09120/) (2026)
- [PatRe: A Full-Stage Office Action and Rebuttal Generation Benchmark for Patent Examination](https://mlsanity.com/paper/2605.03571/) (2026)

## Abstract (arXiv)

Innovation is a major driver of economic and social development, and information about many kinds of innovation is embedded in semi-structured data from patents and patent applications. Although the impact and novelty of innovations expressed in patent data are difficult to measure through traditional means, ML offers a promising set of techniques for evaluating novelty, summarizing contributions, and embedding semantics. In this paper, we introduce the Harvard USPTO Patent Dataset (HUPD), a large-scale, well-structured, and multi-purpose corpus of English-language patent applications filed to the United States Patent and Trademark Office (USPTO) between 2004 and 2018. With more than 4.5 million patent documents, HUPD is two to three times larger than comparable corpora. Unlike previously proposed patent datasets in NLP, HUPD contains the inventor-submitted versions of patent applications--not the final versions of granted patents--thereby allowing us to study patentability at the time of filing using NLP methods for the first time. It is also novel in its inclusion of rich structured metadata alongside the text of patent filings: By providing each application's metadata along with all of its text fields, the dataset enables researchers to perform new sets of NLP tasks that leverage variation in structured covariates. As a case study on the types of research HUPD makes possible, we introduce a new task to the NLP community--namely, binary classification of patent decisions. We additionally show the structured metadata provided in the dataset enables us to conduct explicit studies of concept shifts for this task. Finally, we demonstrate how HUPD can be used for three additional tasks: multi-class classification of patent subject areas, language modeling, and summarization.