# HittER: Hierarchical Transformers for Knowledge Graph Embeddings

arXiv: https://arxiv.org/abs/2008.12813 · HTML: https://mlsanity.com/paper/2008.12813/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2020.

## Built on by (most cited first, 30 of 54)

- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion](https://mlsanity.com/paper/2402.02389/) (2024)
- [Link Prediction on N-ary Relational Facts: A Graph-based Approach](https://mlsanity.com/paper/2105.08476/) (2021)
- [LPFormer: An Adaptive Graph Transformer for Link Prediction](https://mlsanity.com/paper/2310.11009/) (2023)
- [MKGL: Mastery of a Three-Word Language](https://mlsanity.com/paper/2410.07526/) (2024)
- [Structure Pretraining and Prompt Tuning for Knowledge Graph Transfer](https://mlsanity.com/paper/2303.03922/) (2023)
- [Representation Learning on Hyper-Relational and Numeric Knowledge Graphs with Transformers](https://mlsanity.com/paper/2305.18256/) (2023)
- [I Know What You Do Not Know: Knowledge Graph Embedding via Co-distillation Learning](https://mlsanity.com/paper/2208.09828/) (2022)
- [Transformer-based Entity Typing in Knowledge Graphs](https://mlsanity.com/paper/2210.11151/) (2022)
- [QuatDE: Dynamic Quaternion Embedding for Knowledge Graph Completion](https://mlsanity.com/paper/2105.09002/) (2021)
- [KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph Completion](https://mlsanity.com/paper/2208.07622/) (2022)
- [Friendly Neighbors: Contextualized Sequence-to-Sequence Link Prediction](https://mlsanity.com/paper/2305.13059/) (2023)
- [Self-attention Presents Low-dimensional Knowledge Graph Embeddings for Link Prediction](https://mlsanity.com/paper/2112.10644/) (2021)
- [Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning](https://mlsanity.com/paper/2307.03591/) (2023)
- [Re-thinking Knowledge Graph Completion Evaluation from an Information Retrieval Perspective](https://mlsanity.com/paper/2205.04105/) (2022)
- [Relphormer: Relational Graph Transformer for Knowledge Graph Representations](https://mlsanity.com/paper/2205.10852/) (2022)
- [Unleashing the Power of Transformer for Graphs](https://mlsanity.com/paper/2202.10581/) (2022)
- [A Retrieve-and-Read Framework for Knowledge Graph Link Prediction](https://mlsanity.com/paper/2212.09724/) (2022)
- [HyCubE: Efficient Knowledge Hypergraph 3D Circular Convolutional Embedding](https://mlsanity.com/paper/2402.08961/) (2024)
- [Joint Pre-training and Local Re-training: Transferable Representation Learning on Multi-source Knowledge Graphs](https://mlsanity.com/paper/2306.02679/) (2023)
- [HyperFormer: Enhancing Entity and Relation Interaction for Hyper-Relational Knowledge Graph Completion](https://mlsanity.com/paper/2308.06512/) (2023)
- [KnowFormer: Revisiting Transformers for Knowledge Graph Reasoning](https://mlsanity.com/paper/2409.12865/) (2024)
- [A Contextualized BERT model for Knowledge Graph Completion](https://mlsanity.com/paper/2412.11016/) (2024)
- [Inductive Relation Prediction from Relational Paths and Context with Hierarchical Transformers](https://mlsanity.com/paper/2304.00215/) (2023)
- [Knowledge Graph Error Detection with Contrastive Confidence Adaption](https://mlsanity.com/paper/2312.12108/) (2023)
- [Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs](https://mlsanity.com/paper/2401.00608/) (2023)
- [A Conversation is Worth A Thousand Recommendations: A Survey of Holistic Conversational Recommender Systems](https://mlsanity.com/paper/2309.07682/) (2023)
- [UniHR: Hierarchical Representation Learning for Unified Knowledge Graph Link Prediction](https://mlsanity.com/paper/2411.07019/) (2024)
- [Efficient Relation-aware Neighborhood Aggregation in Graph Neural Networks via Tensor Decomposition](https://mlsanity.com/paper/2212.05581/) (2022)
- [Pre-training Transformers for Knowledge Graph Completion](https://mlsanity.com/paper/2303.15682/) (2023)

## Abstract (arXiv)

This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical Transformer model to jointly learn Entity-relation composition and Relational contextualization based on a source entity's neighborhood. Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block. We further design a masked entity prediction task to balance information from the relational context and the source entity itself. Experimental results show that HittER achieves new state-of-the-art results on multiple link prediction datasets. We additionally propose a simple approach to integrate HittER into BERT and demonstrate its effectiveness on two Freebase factoid question answering datasets.