# Preserving Node Distinctness in Graph Autoencoders via Similarity Distillation

arXiv: https://arxiv.org/abs/2406.17517 · HTML: https://mlsanity.com/paper/2406.17517/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 23 of 23)

- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [Graph Self-Supervised Learning: A Survey](https://mlsanity.com/paper/2103.00111/) (2021)
- [Graph Contrastive Learning Automated](https://mlsanity.com/paper/2106.07594/) (2021)
- [A Survey on Oversmoothing in Graph Neural Networks](https://mlsanity.com/paper/2303.10993/) (2023)
- [GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner](https://mlsanity.com/paper/2304.04779/) (2023)
- [InfoGCL: Information-Aware Graph Contrastive Learning](https://mlsanity.com/paper/2110.15438/) (2021)
- [What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders](https://mlsanity.com/paper/2205.10053/) (2022)
- [Semi-Implicit Graph Variational Auto-Encoders](https://mlsanity.com/paper/1908.07078/) (2019)
- [Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules](https://mlsanity.com/paper/2310.14753/) (2023)
- [Graph Attention Auto-Encoders](https://mlsanity.com/paper/1905.10715/) (2019)
- [Knowledge Distillation on Graphs: A Survey](https://mlsanity.com/paper/2302.00219/) (2023)
- [Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology](https://mlsanity.com/paper/1809.03006/) (2018)
- [Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework](https://mlsanity.com/paper/2305.10758/) (2023)
- [SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Masking](https://mlsanity.com/paper/2301.12458/) (2023)
- [Masked Graph Autoencoder with Non-discrete Bandwidths](https://mlsanity.com/paper/2402.03814/) (2024)
- [Exploring Task Unification in Graph Representation Learning via Generative Approach](https://mlsanity.com/paper/2403.14340/) (2024)
- [Refining Latent Representations: A Generative SSL Approach for Heterogeneous Graph Learning](https://mlsanity.com/paper/2310.11102/) (2023)

## Abstract (arXiv)

Graph autoencoders (GAEs), as a kind of generative self-supervised learning approach, have shown great potential in recent years. GAEs typically rely on distance-based criteria, such as mean-square-error (MSE), to reconstruct the input graph. However, relying solely on a single reconstruction criterion may lead to a loss of distinctiveness in the reconstructed graph, causing nodes to collapse into similar representations and resulting in sub-optimal performance. To address this issue, we have developed a simple yet effective strategy to preserve the necessary distinctness in the reconstructed graph. Inspired by the knowledge distillation technique, we found that the dual encoder-decoder architecture of GAEs can be viewed as a teacher-student relationship. Therefore, we propose transferring the knowledge of distinctness from the raw graph to the reconstructed graph, achieved through a simple KL constraint. Specifically, we compute pairwise node similarity scores in the raw graph and reconstructed graph. During the training process, the KL constraint is optimized alongside the reconstruction criterion. We conducted extensive experiments across three types of graph tasks, demonstrating the effectiveness and generality of our strategy. This indicates that the proposed approach can be employed as a plug-and-play method to avoid vague reconstructions and enhance overall performance.