# Representation Learning on Hyper-Relational and Numeric Knowledge Graphs with Transformers

arXiv: https://arxiv.org/abs/2305.18256 · HTML: https://mlsanity.com/paper/2305.18256/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [MMKG: Multi-Modal Knowledge Graphs](https://mlsanity.com/paper/1903.05485/) (2019)
- [Message Passing for Hyper-Relational Knowledge Graphs](https://mlsanity.com/paper/2009.10847/) (2020)
- [On the representation and embedding of knowledge bases beyond binary relations](https://mlsanity.com/paper/1604.08642/) (2016)
- [HittER: Hierarchical Transformers for Knowledge Graph Embeddings](https://mlsanity.com/paper/2008.12813/) (2020)
- [Generalizing Tensor Decomposition for N-ary Relational Knowledge Bases](https://mlsanity.com/paper/2007.03988/) (2020)
- [Embedding Multimodal Relational Data for Knowledge Base Completion](https://mlsanity.com/paper/1809.01341/) (2018)
- [A Survey on Knowledge Graph Embeddings with Literals: Which model links better Literal-ly?](https://mlsanity.com/paper/1910.12507/) (2019)
- [Incorporating Literals into Knowledge Graph Embeddings](https://mlsanity.com/paper/1802.00934/) (2018)
- [Role-Aware Modeling for N-ary Relational Knowledge Bases](https://mlsanity.com/paper/2104.09780/) (2021)
- [Link Prediction on N-ary Relational Facts: A Graph-based Approach](https://mlsanity.com/paper/2105.08476/) (2021)
- [KBLRN : End-to-End Learning of Knowledge Base Representations with Latent, Relational, and Numerical Features](https://mlsanity.com/paper/1709.04676/) (2017)
- [Query Embedding on Hyper-relational Knowledge Graphs](https://mlsanity.com/paper/2106.08166/) (2021)
- [Improving Inductive Link Prediction Using Hyper-Relational Facts](https://mlsanity.com/paper/2107.04894/) (2021)
- [Improving Hyper-Relational Knowledge Graph Completion](https://mlsanity.com/paper/2104.08167/) (2021)
- [SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs](https://mlsanity.com/paper/2110.14890/) (2021)
- [Link Prediction on N-ary Relational Data Based on Relatedness Evaluation](https://mlsanity.com/paper/2104.10424/) (2021)
- [Multi-task Neural Network for Non-discrete Attribute Prediction in Knowledge Graphs](https://mlsanity.com/paper/1708.04828/) (2017)
- [Searching to Sparsify Tensor Decomposition for N-ary Relational Data](https://mlsanity.com/paper/2104.10625/) (2021)
- [Node Attribute Completion in Knowledge Graphs with Multi-Relational Propagation](https://mlsanity.com/paper/2011.05301/) (2020)
- [Learning Representations of Bi-level Knowledge Graphs for Reasoning beyond Link Prediction](https://mlsanity.com/paper/2302.02601/) (2023)

## Built on by (most cited first, 16 of 16)

- [InGram: Inductive Knowledge Graph Embedding via Relation Graphs](https://mlsanity.com/paper/2305.19987/) (2023)
- [SpoT-Mamba: Learning Long-Range Dependency on Spatio-Temporal Graphs with Selective State Spaces](https://mlsanity.com/paper/2406.11244/) (2024)
- [Dynamic Relation-Attentive Graph Neural Networks for Fraud Detection](https://mlsanity.com/paper/2310.04171/) (2023)
- [UniHR: Hierarchical Representation Learning for Unified Knowledge Graph Link Prediction](https://mlsanity.com/paper/2411.07019/) (2024)
- [Inductive Link Prediction on N-ary Relational Facts via Semantic Hypergraph Reasoning](https://mlsanity.com/paper/2503.20676/) (2025)
- [HyperMono: A Monotonicity-aware Approach to Hyper-Relational Knowledge Representation](https://mlsanity.com/paper/2404.09848/) (2024)
- [PAC-Bayesian Generalization Bounds for Knowledge Graph Representation Learning](https://mlsanity.com/paper/2405.06418/) (2024)
- [Generalizing Hyperedge Expansion for Hyper-relational Knowledge Graph Modeling](https://mlsanity.com/paper/2411.06191/) (2024)
- [SAIF: A Comprehensive Framework for Evaluating the Risks of Generative AI in the Public Sector](https://mlsanity.com/paper/2501.08814/) (2025)
- [CHAINSFORMER: Numerical Reasoning on Knowledge Graphs from a Chain Perspective](https://mlsanity.com/paper/2504.14282/) (2025)
- [Two-dimensional Taxonomy for N-ary Knowledge Representation Learning Methods](https://mlsanity.com/paper/2506.05626/) (2025)
- [AtGCN: A Graph Convolutional Network For Ataxic Gait Detection](https://mlsanity.com/paper/2410.22862/) (2024)
- [VITA: Versatile Time Representation Learning for Temporal Hyper-Relational Knowledge Graphs](https://mlsanity.com/paper/2505.11803/) (2025)
- [HEHRGNN: A Unified Embedding Model for Knowledge Graphs with Hyperedges and Hyper-Relational Edges](https://mlsanity.com/paper/2602.18897/) (2026)
- [AlertStar: Path-Aware Alert Prediction on Hyper-Relational Knowledge Graphs](https://mlsanity.com/paper/2604.03104/) (2026)
- [Generative Representation Learning on Hyper-relational Knowledge Graphs via Masked Discrete Diffusion](https://mlsanity.com/paper/2605.24064/) (2026)

## Abstract (arXiv)

A hyper-relational knowledge graph has been recently studied where a triplet is associated with a set of qualifiers; a qualifier is composed of a relation and an entity, providing auxiliary information for a triplet. While existing hyper-relational knowledge graph embedding methods assume that the entities are discrete objects, some information should be represented using numeric values, e.g., (J.R.R., was born in, 1892). Also, a triplet (J.R.R., educated at, Oxford Univ.) can be associated with a qualifier such as (start time, 1911). In this paper, we propose a unified framework named HyNT that learns representations of a hyper-relational knowledge graph containing numeric literals in either triplets or qualifiers. We define a context transformer and a prediction transformer to learn the representations based not only on the correlations between a triplet and its qualifiers but also on the numeric information. By learning compact representations of triplets and qualifiers and feeding them into the transformers, we reduce the computation cost of using transformers. Using HyNT, we can predict missing numeric values in addition to missing entities or relations in a hyper-relational knowledge graph. Experimental results show that HyNT significantly outperforms state-of-the-art methods on real-world datasets.