# HINormer: Representation Learning On Heterogeneous Information Networks with Graph Transformer

arXiv: https://arxiv.org/abs/2302.11329 · HTML: https://mlsanity.com/paper/2302.11329/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 64 references that are in the map.
- First posted on arXiv 22 Feb 2023.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [Hierarchical Graph Representation Learning with Differentiable Pooling](https://mlsanity.com/paper/1806.08804/) (2018)
- [Heterogeneous Graph Attention Network](https://mlsanity.com/paper/1903.07293/) (2019)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [DropEdge: Towards Deep Graph Convolutional Networks on Node Classification](https://mlsanity.com/paper/1907.10903/) (2019)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [On the Bottleneck of Graph Neural Networks and its Practical Implications](https://mlsanity.com/paper/2006.05205/) (2020)
- [LINE: Large-scale Information Network Embedding](https://mlsanity.com/paper/1503.03578/) (2015)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View](https://mlsanity.com/paper/1909.03211/) (2019)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [Rethinking Graph Transformers with Spectral Attention](https://mlsanity.com/paper/2106.03893/) (2021)
- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding](https://mlsanity.com/paper/2002.01680/) (2020)
- [PairNorm: Tackling Oversmoothing in GNNs](https://mlsanity.com/paper/1909.12223/) (2019)

## Built on by (most cited first, 30 of 35)

- [A Comprehensive Survey on Deep Graph Representation Learning](https://mlsanity.com/paper/2304.05055/) (2023)
- [Relational Graph Transformer](https://mlsanity.com/paper/2505.10960/) (2025)
- [Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining](https://mlsanity.com/paper/2403.04780/) (2024)
- [Efficient Heterogeneous Graph Learning via Random Projection](https://mlsanity.com/paper/2310.14481/) (2023)
- [Spectral Heterogeneous Graph Convolutions via Positive Noncommutative Polynomials](https://mlsanity.com/paper/2305.19872/) (2023)
- [Few-Shot Causal Representation Learning for Out-of-Distribution Generalization on Heterogeneous Graphs](https://mlsanity.com/paper/2401.03597/) (2024)
- [HAGNN: Hybrid Aggregation for Heterogeneous Graph Neural Networks](https://mlsanity.com/paper/2307.01636/) (2023)
- [Long-range Meta-path Search on Large-scale Heterogeneous Graphs](https://mlsanity.com/paper/2307.08430/) (2023)
- [Hetero2Net: Heterophily-aware Representation Learning on Heterogenerous Graphs](https://mlsanity.com/paper/2310.11664/) (2023)
- [RSEA-MVGNN: Multi-View Graph Neural Network with Reliable Structural Enhancement and Aggregation](https://mlsanity.com/paper/2408.07331/) (2024)
- [HGMP:Heterogeneous Graph Multi-Task Prompt Learning](https://mlsanity.com/paper/2507.07405/) (2025)
- [Integrating Temporal and Structural Context in Graph Transformers for Relational Deep Learning](https://mlsanity.com/paper/2511.04557/) (2025)
- [HeteGraph-Mamba: Heterogeneous Graph Learning via Selective State Space Model](https://mlsanity.com/paper/2405.13915/) (2024)
- [Adaptive Heterogeneous Graph Neural Networks: Bridging Heterophily and Heterogeneity](https://mlsanity.com/paper/2508.06034/) (2025)
- [HHGT: Hierarchical Heterogeneous Graph Transformer for Heterogeneous Graph Representation Learning](https://mlsanity.com/paper/2407.13158/) (2024)
- [ScaleGNN: Towards Scalable Graph Neural Networks via Adaptive High-order Neighboring Feature Fusion](https://mlsanity.com/paper/2504.15920/) (2025)
- [Heterogeneous Temporal Hypergraph Neural Network](https://mlsanity.com/paper/2506.17312/) (2025)
- [Towards Efficient LLM-aware Heterogeneous Graph Learning](https://mlsanity.com/paper/2511.17923/) (2025)
- [Variational Graph Autoencoder for Heterogeneous Information Networks with Missing and Inaccurate Attributes](https://mlsanity.com/paper/2311.07929/) (2023)
- [Heterogeneous Graph Neural Networks with Loss-decrease-aware Curriculum Learning](https://mlsanity.com/paper/2405.06522/) (2024)
- [Addressing Graph Heterogeneity and Heterophily from A Spectral Perspective](https://mlsanity.com/paper/2410.13373/) (2024)
- [Power Grid Control with Graph-Based Distributed Reinforcement Learning](https://mlsanity.com/paper/2509.02861/) (2025)
- [Echoless Label-Based Pre-computation for Memory-Efficient Heterogeneous Graph Learning](https://mlsanity.com/paper/2511.11081/) (2025)
- [Spiking Heterogeneous Graph Attention Networks](https://mlsanity.com/paper/2601.02401/) (2025)
- [HetCAN: A Heterogeneous Graph Cascade Attention Network with Dual-Level Awareness](https://mlsanity.com/paper/2311.03275/) (2023)
- [AGHINT: Attribute-Guided Representation Learning on Heterogeneous Information Networks with Transformer](https://mlsanity.com/paper/2404.10443/) (2024)
- [Perturbation Ontology based Graph Attention Networks](https://mlsanity.com/paper/2411.18520/) (2024)
- [Academic Network Representation via Prediction-Sampling Incorporated Tensor Factorization](https://mlsanity.com/paper/2504.08323/) (2025)
- [Are Heterogeneous Graph Neural Networks Truly Effective for Node Classification? A Causal Perspective](https://mlsanity.com/paper/2510.05750/) (2025)
- [HeSRN: Representation Learning On Heterogeneous Graphs via Slot-Aware Retentive Network](https://mlsanity.com/paper/2510.09767/) (2025)

## Abstract (arXiv)

Recent studies have highlighted the limitations of message-passing based graph neural networks (GNNs), e.g., limited model expressiveness, over-smoothing, over-squashing, etc. To alleviate these issues, Graph Transformers (GTs) have been proposed which work in the paradigm that allows message passing to a larger coverage even across the whole graph. Hinging on the global range attention mechanism, GTs have shown a superpower for representation learning on homogeneous graphs. However, the investigation of GTs on heterogeneous information networks (HINs) is still under-exploited. In particular, on account of the existence of heterogeneity, HINs show distinct data characteristics and thus require different treatment. To bridge this gap, in this paper we investigate the representation learning on HINs with Graph Transformer, and propose a novel model named HINormer, which capitalizes on a larger-range aggregation mechanism for node representation learning. In particular, assisted by two major modules, i.e., a local structure encoder and a heterogeneous relation encoder, HINormer can capture both the structural and heterogeneous information of nodes on HINs for comprehensive node representations. We conduct extensive experiments on four HIN benchmark datasets, which demonstrate that our proposed model can outperform the state-of-the-art.