# Position-based Hash Embeddings For Scaling Graph Neural Networks

arXiv: https://arxiv.org/abs/2109.00101 · HTML: https://mlsanity.com/paper/2109.00101/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 14 of 14)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [RGCNN: Regularized Graph CNN for Point Cloud Segmentation](https://mlsanity.com/paper/1806.02952/) (2018)
- [Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems](https://mlsanity.com/paper/1909.02107/) (2019)
- [Hash Embeddings for Efficient Word Representations](https://mlsanity.com/paper/1709.03933/) (2017)
- [Model Size Reduction Using Frequency Based Double Hashing for Recommender Systems](https://mlsanity.com/paper/2007.14523/) (2020)
- [Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks](https://mlsanity.com/paper/1804.08313/) (2018)
- [Heterogeneous Molecular Graph Neural Networks for Predicting Molecule Properties](https://mlsanity.com/paper/2009.12710/) (2020)
- [Candidate Generation with Binary Codes for Large-Scale Top-N Recommendation](https://mlsanity.com/paper/1909.05475/) (2019)
- [Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks](https://mlsanity.com/paper/1706.03993/) (2017)

## Built on by (most cited first, 3 of 3)

- [Nimble GNN Embedding with Tensor-Train Decomposition](https://mlsanity.com/paper/2206.10581/) (2022)
- [Injecting Domain Knowledge from Empirical Interatomic Potentials to Neural Networks for Predicting Material Properties](https://mlsanity.com/paper/2210.08047/) (2022)
- [Scaling Graph Neural Networks for Friend Recommendation: Multi-Hash User Embeddings and Temporal Neighbor Sampling](https://mlsanity.com/paper/2608.27413/) (2026)

## Abstract (arXiv)

Graph Neural Networks (GNNs) bring the power of deep representation learning to graph and relational data and achieve state-of-the-art performance in many applications. GNNs compute node representations by taking into account the topology of the node's ego-network and the features of the ego-network's nodes. When the nodes do not have high-quality features, GNNs learn an embedding layer to compute node embeddings and use them as input features. However, the size of the embedding layer is linear to the product of the number of nodes in the graph and the dimensionality of the embedding and does not scale to big data and graphs with hundreds of millions of nodes. To reduce the memory associated with this embedding layer, hashing-based approaches, commonly used in applications like NLP and recommender systems, can potentially be used. However, a direct application of these ideas fails to exploit the fact that in many real-world graphs, nodes that are topologically close will tend to be related to each other (homophily) and as such their representations will be similar. In this work, we present approaches that take advantage of the nodes' position in the graph to dramatically reduce the memory required, with minimal if any degradation in the quality of the resulting GNN model. Our approaches decompose a node's embedding into two components: a position-specific component and a node-specific component. The position-specific component models homophily and the node-specific component models the node-to-node variation. Extensive experiments using different datasets and GNN models show that our methods are able to reduce the memory requirements by 88% to 97% while achieving, in nearly all cases, better classification accuracy than other competing approaches, including the full embeddings.