# Heterogeneous Dependency Graph-Guided Attentionfor Patent Representation Learning

arXiv: https://arxiv.org/abs/2605.10073 · HTML: https://mlsanity.com/paper/2605.10073/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 11 May 2026.

## Builds on (most cited first, 10 of 10)

- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [The Harvard USPTO Patent Dataset: A Large-Scale, Well-Structured, and Multi-Purpose Corpus of Patent Applications](https://mlsanity.com/paper/2207.04043/) (2022)
- [PatentSBERTa: A Deep NLP based Hybrid Model for Patent Distance and Classification using Augmented SBERT](https://mlsanity.com/paper/2103.11933/) (2021)
- [PaECTER: Patent-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2402.19411/) (2024)
- [Linguistically Informed Masking for Representation Learning in the Patent Domain](https://mlsanity.com/paper/2106.05768/) (2021)
- [PatentMind: A Multi-Aspect Reasoning Graph for Patent Similarity Evaluation](https://mlsanity.com/paper/2505.19347/) (2025)
- [PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding](https://mlsanity.com/paper/2510.22264/) (2025)
- [Beyond Scaling: Predicting Patent Approval with Domain-specific Fine-grained Claim Dependency Graph](https://mlsanity.com/paper/2404.14372/) (2024)
- [Patent Language Model Pretraining with ModernBERT](https://mlsanity.com/paper/2509.14926/) (2025)
- [Patent Representation Learning via Self-supervision](https://mlsanity.com/paper/2511.10657/) (2025)

## Abstract (arXiv)

Pre-trained language models advance patent classification and retrieval by encoding claims as flat token sequences, but they overlook the dependency hierarchy among claims. Incorporating this hierarchy into self-attention poses two challenges. First, claim dependencies include relation types with different levels of reliability, so treating them uniformly may allow noisy technical relations to interfere with more reliable legal citations. Second, claim dependencies are defined at the claim level, whereas Transformer attention operates over tokens, making direct structural injection nontrivial. We propose the Patent Heterogeneous Attention-Guided Graph Encoder (PHAGE), which constructs a typed claim graph that distinguishes legal citations from technical relations. PHAGE projects this claim-level topology into token-level attention through a connectivity mask and learnable relation-aware biases, and fine-tunes the encoder using a dual-granularity contrastive objective that combines inter-patent taxonomy with intra-patent topology. At inference, the graph-specific attention components are removed, allowing representations to be generated through a standard encoder forward pass without CDG construction. Experiments on patent classification, retrieval, and clustering show that PHAGE consistently outperforms domain-adapted and citation-aware baselines, demonstrating the value of claim-level structural guidance for graph-free patent representation learning.