# Large-Scale Network Embedding in Apache Spark

arXiv: https://arxiv.org/abs/2106.10620 · HTML: https://mlsanity.com/paper/2106.10620/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 20 Jun 2021.

## Builds on (most cited first, 21 of 21)

- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Representation Learning on Graphs: Methods and Applications](https://mlsanity.com/paper/1709.05584/) (2017)
- [A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications](https://mlsanity.com/paper/1709.07604/) (2017)
- [Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2vec](https://mlsanity.com/paper/1710.02971/) (2017)
- [Graph Embedding Techniques, Applications, and Performance: A Survey](https://mlsanity.com/paper/1705.02801/) (2017)
- [AliGraph: A Comprehensive Graph Neural Network Platform](https://mlsanity.com/paper/1902.08730/) (2019)
- [A Survey on Network Embedding](https://mlsanity.com/paper/1711.08752/) (2017)
- [PyTorch-BigGraph: A Large-scale Graph Embedding System](https://mlsanity.com/paper/1903.12287/) (2019)
- [GraphZoom: A multi-level spectral approach for accurate and scalable graph embedding](https://mlsanity.com/paper/1910.02370/) (2019)
- [NetSMF: Large-Scale Network Embedding as Sparse Matrix Factorization](https://mlsanity.com/paper/1906.11156/) (2019)
- [Accurate, Efficient and Scalable Graph Embedding](https://mlsanity.com/paper/1810.11899/) (2018)
- [Is a Single Vector Enough? Exploring Node Polysemy for Network Embedding](https://mlsanity.com/paper/1905.10668/) (2019)
- [Is a Single Embedding Enough? Learning Node Representations that Capture Multiple Social Contexts](https://mlsanity.com/paper/1905.02138/) (2019)
- [Initialization for Network Embedding: A Graph Partition Approach](https://mlsanity.com/paper/1908.10697/) (2019)
- [Distributed Algorithms for Fully Personalized PageRank on Large Graphs](https://mlsanity.com/paper/1903.11749/) (2019)
- [SepNE: Bringing Separability to Network Embedding](https://mlsanity.com/paper/1811.05614/) (2018)
- [Spectral Network Embedding: A Fast and Scalable Method via Sparsity](https://mlsanity.com/paper/1806.02623/) (2018)
- [A Survey on Spark Ecosystem for Big Data Processing](https://mlsanity.com/paper/1811.08834/) (2018)

## Built on by (most cited first, 6 of 6)

- [Graph Unlearning with Efficient Partial Retraining](https://mlsanity.com/paper/2403.07353/) (2024)
- [Collaborative Graph Neural Networks for Attributed Network Embedding](https://mlsanity.com/paper/2307.11981/) (2023)
- [Permutation-Invariant Graph Partitioning:How Graph Neural Networks Capture Structural Interactions?](https://mlsanity.com/paper/2312.08671/) (2023)
- [LPS-GNN : Deploying Graph Neural Networks on Graphs with 100-Billion Edges](https://mlsanity.com/paper/2507.14570/) (2025)
- [FROG: Effective Friend Recommendation in Online Games via Modality-aware User Preferences](https://mlsanity.com/paper/2504.09428/) (2025)
- [Heterogeneous Influence Maximization in User Recommendation](https://mlsanity.com/paper/2508.13517/) (2025)

## Abstract (arXiv)

Network embedding has been widely used in social recommendation and network analysis, such as recommendation systems and anomaly detection with graphs. However, most of previous approaches cannot handle large graphs efficiently, due to that (i) computation on graphs is often costly and (ii) the size of graph or the intermediate results of vectors could be prohibitively large, rendering it difficult to be processed on a single machine. In this paper, we propose an efficient and effective distributed algorithm for network embedding on large graphs using Apache Spark, which recursively partitions a graph into several small-sized subgraphs to capture the internal and external structural information of nodes, and then computes the network embedding for each subgraph in parallel. Finally, by aggregating the outputs on all subgraphs, we obtain the embeddings of nodes in a linear cost. After that, we demonstrate in various experiments that our proposed approach is able to handle graphs with billions of edges within a few hours and is at least 4 times faster than the state-of-the-art approaches. Besides, it achieves up to 4.25% and 4.27% improvements on link prediction and node classification tasks respectively. In the end, we deploy the proposed algorithms in two online games of Tencent with the applications of friend recommendation and item recommendation, which improve the competitors by up to 91.11% in running time and up to 12.80% in the corresponding evaluation metrics.