# PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model

arXiv: https://arxiv.org/abs/1906.02124 · HTML: https://mlsanity.com/paper/1906.02124/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2019.

## Built on by (most cited first, 24 of 24)

- [Semantic Similarity Matching for Patent Documents Using Ensemble BERT-related Model and Novel Text Processing Method](https://mlsanity.com/paper/2401.06782/) (2024)
- [Towards General Purpose Medical AI: Continual Learning Medical Foundation Model](https://mlsanity.com/paper/2303.06580/) (2023)
- [Biomedical and Clinical Language Models for Spanish: On the Benefits of Domain-Specific Pretraining in a Mid-Resource Scenario](https://mlsanity.com/paper/2109.03570/) (2021)
- [Linguistically Informed Masking for Representation Learning in the Patent Domain](https://mlsanity.com/paper/2106.05768/) (2021)
- [Reconciling Methodological Paradigms: Employing Large Language Models as Novice Qualitative Research Assistants in Talent Management Research](https://mlsanity.com/paper/2408.11043/) (2024)
- [PatentMind: A Multi-Aspect Reasoning Graph for Patent Similarity Evaluation](https://mlsanity.com/paper/2505.19347/) (2025)
- [G2PTL: A Pre-trained Model for Delivery Address and its Applications in Logistics System](https://mlsanity.com/paper/2304.01559/) (2023)
- [Qualitative Insights Tool (QualIT): LLM Enhanced Topic Modeling](https://mlsanity.com/paper/2409.15626/) (2024)
- [Self-Filtered Distillation with LLMs-generated Trust Indicators for Reliable Patent Classification](https://mlsanity.com/paper/2510.05431/) (2025)
- [Can Language Representation Models Think in Bets?](https://mlsanity.com/paper/2210.07519/) (2022)
- [A Novel Patent Similarity Measurement Methodology: Semantic Distance and Technological Distance](https://mlsanity.com/paper/2303.16767/) (2023)
- [A comparative analysis of embedding models for patent similarity](https://mlsanity.com/paper/2403.16630/) (2024)
- [Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph](https://mlsanity.com/paper/2404.14372/) (2024)
- [PatentEval: Understanding Errors in Patent Generation](https://mlsanity.com/paper/2406.06589/) (2024)
- [Can AI Examine Novelty of Patents?: Novelty Evaluation Based on the Correspondence between Patent Claim and Prior Art](https://mlsanity.com/paper/2502.06316/) (2025)
- [Understanding Prior Bias and Choice Paralysis in Transformer-based Language Representation Models through Four Experimental Probes](https://mlsanity.com/paper/2210.01258/) (2022)
- [Solar cell patent classification method based on keyword extraction and deep neural network](https://mlsanity.com/paper/2109.08796/) (2021)
- [K-12BERT: BERT for K-12 education](https://mlsanity.com/paper/2205.12335/) (2022)
- [Large-Scale Text Analysis Using Generative Language Models: A Case Study in Discovering Public Value Expressions in AI Patents](https://mlsanity.com/paper/2305.10383/) (2023)
- [Introduction of a novel word embedding approach based on technology labels extracted from patent data](https://mlsanity.com/paper/2102.00425/) (2021)
- [Adaptation of domain-specific transformer models with text oversampling for sentiment analysis of social media posts on Covid-19 vaccines](https://mlsanity.com/paper/2209.10966/) (2022)
- [CinPatent: Datasets for Patent Classification](https://mlsanity.com/paper/2212.12192/) (2022)
- [From scratch to silver: Creating trustworthy training data for patent-SDG classification using Large Language Models](https://mlsanity.com/paper/2509.09303/) (2025)
- [Benchmarking Embedding Geometries and Clustering Algorithms for Unsupervised Patent Claim Classification](https://mlsanity.com/paper/2610.12997/) (2026)

## Abstract (arXiv)

In this work we focus on fine-tuning a pre-trained BERT model and applying it to patent classification. When applied to large datasets of over two millions patents, our approach outperforms the state of the art by an approach using CNN with word embeddings. In addition, we focus on patent claims without other parts in patent documents. Our contributions include: (1) a new state-of-the-art method based on pre-trained BERT model and fine-tuning for patent classification, (2) a large dataset USPTO-3M at the CPC subclass level with SQL statements that can be used by future researchers, (3) showing that patent claims alone are sufficient for classification task, in contrast to conventional wisdom.