# TransformerG2G: Adaptive time-stepping for learning temporal graph embeddings using transformers

arXiv: https://arxiv.org/abs/2307.02588 · HTML: https://mlsanity.com/paper/2307.02588/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [LINE: Large-scale Information Network Embedding](https://mlsanity.com/paper/1503.03578/) (2015)
- [Temporal Graph Networks for Deep Learning on Dynamic Graphs](https://mlsanity.com/paper/2006.10637/) (2020)
- [Inductive Representation Learning on Temporal Graphs](https://mlsanity.com/paper/2002.07962/) (2020)
- [Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks](https://mlsanity.com/paper/1908.01207/) (2019)
- [EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs](https://mlsanity.com/paper/1902.10191/) (2019)
- [Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking](https://mlsanity.com/paper/1707.03815/) (2017)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction](https://mlsanity.com/paper/2106.06130/) (2021)
- [Dynamic Graph Convolutional Recurrent Network for Traffic Prediction: Benchmark and Solution](https://mlsanity.com/paper/2104.14917/) (2021)
- [ROLAND: Graph Learning Framework for Dynamic Graphs](https://mlsanity.com/paper/2208.07239/) (2022)
- [Towards Better Evaluation for Dynamic Link Prediction](https://mlsanity.com/paper/2207.10128/) (2022)
- [PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/2108.01938/) (2021)
- [DynGEM: Deep Embedding Method for Dynamic Graphs](https://mlsanity.com/paper/1805.11273/) (2018)
- [dyngraph2vec: Capturing Network Dynamics using Dynamic Graph Representation Learning](https://mlsanity.com/paper/1809.02657/) (2018)
- [Everything is Connected: Graph Neural Networks](https://mlsanity.com/paper/2301.08210/) (2023)
- [Graph Representation Learning in Biomedicine](https://mlsanity.com/paper/2104.04883/) (2021)
- [Temporal Knowledge Graph Forecasting with Neural ODE](https://mlsanity.com/paper/2101.05151/) (2021)
- [Learning continuous-time PDEs from sparse data with graph neural networks](https://mlsanity.com/paper/2006.08956/) (2020)
- [Understanding graph embedding methods and their applications](https://mlsanity.com/paper/2012.08019/) (2020)
- [Dynamic Network Embedding Survey](https://mlsanity.com/paper/2103.15447/) (2021)
- [Modularity-Aware Graph Autoencoders for Joint Community Detection and Link Prediction](https://mlsanity.com/paper/2202.00961/) (2022)
- [A Graph Gaussian Embedding Method for Predicting Alzheimer's Disease Progression with MEG Brain Networks](https://mlsanity.com/paper/2005.05784/) (2020)
- [A Survey on Dynamic Network Embedding](https://mlsanity.com/paper/2006.08093/) (2020)
- [DynG2G: An Efficient Stochastic Graph Embedding Method for Temporal Graphs](https://mlsanity.com/paper/2109.13441/) (2021)

## Built on by (most cited first, 9 of 9)

- [Accelerating Phase Field Simulations Through a Hybrid Adaptive Fourier Neural Operator with U-Net Backbone](https://mlsanity.com/paper/2406.17119/) (2024)
- [A Comparative Study on Dynamic Graph Embedding based on Mamba and Transformers](https://mlsanity.com/paper/2412.11293/) (2024)
- [Automatic selection of the best neural architecture for time series forecasting](https://mlsanity.com/paper/2501.12215/) (2025)
- [Frugal Machine Learning for Energy-efficient, and Resource-aware Artificial Intelligence](https://mlsanity.com/paper/2506.01869/) (2025)
- [BrainATCL: Adaptive Temporal Brain Connectivity Learning for Functional Link Prediction and Age Estimation](https://mlsanity.com/paper/2508.07106/) (2025)
- [DyGSSM: Multi-view Dynamic Graph Embeddings with State Space Model Gradient Update](https://mlsanity.com/paper/2505.09017/) (2025)
- [Process-Informed Forecasting of Complex Thermal Dynamics in Pharmaceutical Manufacturing](https://mlsanity.com/paper/2509.20349/) (2025)
- [DynaSTy: A Framework for SpatioTemporal Node Attribute Prediction in Dynamic Graphs](https://mlsanity.com/paper/2601.05391/) (2026)
- [AlertStar: Path-Aware Alert Prediction on Hyper-Relational Knowledge Graphs](https://mlsanity.com/paper/2604.03104/) (2026)

## Abstract (arXiv)

Dynamic graph embedding has emerged as a very effective technique for addressing diverse temporal graph analytic tasks (i.e., link prediction, node classification, recommender systems, anomaly detection, and graph generation) in various applications. Such temporal graphs exhibit heterogeneous transient dynamics, varying time intervals, and highly evolving node features throughout their evolution. Hence, incorporating long-range dependencies from the historical graph context plays a crucial role in accurately learning their temporal dynamics. In this paper, we develop a graph embedding model with uncertainty quantification, TransformerG2G, by exploiting the advanced transformer encoder to first learn intermediate node representations from its current state (t) and previous context (over timestamps [t-1, t-l], l is the length of context). Moreover, we employ two projection layers to generate lower-dimensional multivariate Gaussian distributions as each node's latent embedding at timestamp t. We consider diverse benchmarks with varying levels of ``novelty" as measured by the TEA (Temporal Edge Appearance) plots. Our experiments demonstrate that the proposed TransformerG2G model outperforms conventional multi-step methods and our prior work (DynG2G) in terms of both link prediction accuracy and computational efficiency, especially for high degree of novelty. Furthermore, the learned time-dependent attention weights across multiple graph snapshots reveal the development of an automatic adaptive time stepping enabled by the transformer. Importantly, by examining the attention weights, we can uncover temporal dependencies, identify influential elements, and gain insights into the complex interactions within the graph structure. For example, we identified a strong correlation between attention weights and node degree at the various stages of the graph topology evolution.