# PaECTER: Patent-level Representation Learning using Citation-informed Transformers

arXiv: https://arxiv.org/abs/2402.19411 · HTML: https://mlsanity.com/paper/2402.19411/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)

## Built on by (most cited first, 11 of 11)

- [PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding](https://mlsanity.com/paper/2510.22264/) (2025)
- [Missing vs. Unused Knowledge Hypothesis for Language Model Bottlenecks in Patent Understanding](https://mlsanity.com/paper/2505.12452/) (2025)
- [Citation-Driven Multi-View Training for Patent Embeddings: QaECTER and Sophia-Bench](https://mlsanity.com/paper/2604.22897/) (2026)
- [A Hybrid Ai Framework For Strategic Patent Portfolio Pruning: Integrating Learning To-Rank And Market Need Analysis For Technology Transfer Optimization](https://mlsanity.com/paper/2509.00958/) (2025)
- [From scratch to silver: Creating trustworthy training data for patent-SDG classification using Large Language Models](https://mlsanity.com/paper/2509.09303/) (2025)
- [Adaptive Multi-Stage Patent Claim Generation with Unified Quality Assessment](https://mlsanity.com/paper/2601.09120/) (2026)
- [Anticipating Innovation Using Large Language Models](https://mlsanity.com/paper/2605.04875/) (2026)
- [Heterogeneous Dependency Graph-Guided Attentionfor Patent Representation Learning](https://mlsanity.com/paper/2605.10073/) (2026)
- [When Does Synthetic Patent Data Help? Volume-Fidelity Trade-offs in Low-Resource Multi-Label Classification](https://mlsanity.com/paper/2605.24296/) (2026)
- [From Rules to Neural Graphs: Scalable Structured Prediction for Patent Prior Art Search](https://mlsanity.com/paper/2610.01553/) (2026)
- [SciTBERT: A family of chronologically consistent language models for scientific and technological language processing](https://mlsanity.com/paper/2610.12207/) (2026)

## Abstract (arXiv)

PaECTER is an open-source document-level encoder specific for patents. We fine-tune BERT for Patents with examiner-added citation information to generate numerical representations for patent documents. PaECTER performs better in similarity tasks than current state-of-the-art models used in the patent domain. More specifically, our model outperforms the patent specific pre-trained language model (BERT for Patents) and general-purpose text embedding models (e.g., E5, GTE, and BGE) on our patent citation prediction test dataset on different rank evaluation metrics. PaECTER predicts at least one most similar patent at a rank of 1.32 on average when compared against 25 irrelevant patents. Numerical representations generated by PaECTER from patent text can be used for downstream tasks such as classification, tracing knowledge flows, or semantic similarity search. Semantic similarity search is especially relevant in the context of prior art search for both inventors and patent examiners.