# Spectral Feature Augmentation for Graph Contrastive Learning and Beyond

arXiv: https://arxiv.org/abs/2212.01026 · HTML: https://mlsanity.com/paper/2212.01026/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 84 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 30 of 50)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Graph Contrastive Learning Automated](https://mlsanity.com/paper/2106.07594/) (2021)
- [Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks](https://mlsanity.com/paper/2007.02901/) (2020)
- [Graph Representation Learning via Graphical Mutual Information Maximization](https://mlsanity.com/paper/2002.01169/) (2020)
- [Adversarial Graph Augmentation to Improve Graph Contrastive Learning](https://mlsanity.com/paper/2106.05819/) (2021)
- [SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation](https://mlsanity.com/paper/2202.03104/) (2022)
- [Dataset Augmentation in Feature Space](https://mlsanity.com/paper/1702.05538/) (2017)
- [Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning](https://mlsanity.com/paper/2108.01775/) (2021)
- [An Empirical Study of Graph Contrastive Learning](https://mlsanity.com/paper/2109.01116/) (2021)

## Built on by (most cited first, 30 of 49)

- [3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition](https://mlsanity.com/paper/2303.14474/) (2023)
- [A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy](https://mlsanity.com/paper/2302.10637/) (2023)
- [Hyperbolic Representation Learning: Revisiting and Advancing](https://mlsanity.com/paper/2306.09118/) (2023)
- [SpeGCL: Self-supervised Graph Spectrum Contrastive Learning without Positive Samples](https://mlsanity.com/paper/2410.10365/) (2024)
- [Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network](https://mlsanity.com/paper/2412.15678/) (2024)
- [kHGCN: Tree-likeness Modeling via Continuous and Discrete Curvature Learning](https://mlsanity.com/paper/2212.01793/) (2022)
- [Traffic Forecasting on New Roads Using Spatial Contrastive Pre-Training (SCPT)](https://mlsanity.com/paper/2305.05237/) (2023)
- [Unveiling Vulnerabilities of Contrastive Recommender Systems to Poisoning Attacks](https://mlsanity.com/paper/2311.18244/) (2023)
- [Meet JEANIE: a Similarity Measure for 3D Skeleton Sequences via Temporal-Viewpoint Alignment](https://mlsanity.com/paper/2402.04599/) (2024)
- [Because Every Sensor Is Unique, so Is Every Pair: Handling Dynamicity in Traffic Forecasting](https://mlsanity.com/paper/2302.09956/) (2023)
- [Modulate Your Spectrum in Self-Supervised Learning](https://mlsanity.com/paper/2305.16789/) (2023)
- [Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning](https://mlsanity.com/paper/2310.18209/) (2023)
- [Deep Structural Knowledge Exploitation and Synergy for Estimating Node Importance Value on Heterogeneous Information Networks](https://mlsanity.com/paper/2402.12411/) (2024)
- [Architecture Matters: Uncovering Implicit Mechanisms in Graph Contrastive Learning](https://mlsanity.com/paper/2311.02687/) (2023)
- [ADEdgeDrop: Adversarial Edge Dropping for Robust Graph Neural Networks](https://mlsanity.com/paper/2403.09171/) (2024)
- [Hypergraph Self-supervised Learning with Sampling-efficient Signals](https://mlsanity.com/paper/2404.11825/) (2024)
- [Learning Partial Correlation based Deep Visual Representation for Image Classification](https://mlsanity.com/paper/2304.11597/) (2023)
- [GraphTOP: Graph Topology-Oriented Prompting for Graph Neural Networks](https://mlsanity.com/paper/2510.22451/) (2025)
- [Spectral-Aware Augmentation for Enhanced Graph Representation Learning](https://mlsanity.com/paper/2310.13845/) (2023)
- [Do Neural Scaling Laws Exist on Graph Self-Supervised Learning?](https://mlsanity.com/paper/2408.11243/) (2024)
- [Overcoming Pitfalls in Graph Contrastive Learning Evaluation: Toward Comprehensive Benchmarks](https://mlsanity.com/paper/2402.15680/) (2024)
- [Text as Any-Modality for Zero-Shot Classification by Consistent Prompt Tuning](https://mlsanity.com/paper/2508.06382/) (2025)
- [CogniVerse: Revolutionizing Multi-Modal Retrieval-Augmented Generation with Cognitive Reflection and Geometric Reasoning](https://mlsanity.com/paper/2605.29602/) (2026)
- [Pseudo Contrastive Learning for Graph-based Semi-supervised Learning](https://mlsanity.com/paper/2302.09532/) (2023)
- [Exploiting Field Dependencies for Learning on Categorical Data](https://mlsanity.com/paper/2307.09321/) (2023)
- [Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalance](https://mlsanity.com/paper/2402.11742/) (2024)
- [EEG-MACS: Manifold Attention and Confidence Stratification for EEG-based Cross-Center Brain Disease Diagnosis under Unreliable Annotations](https://mlsanity.com/paper/2405.00734/) (2024)
- [FedHGN: A Federated Framework for Heterogeneous Graph Neural Networks](https://mlsanity.com/paper/2305.09729/) (2023)
- [Graph Contrastive Learning with Low-Rank Regularization and Low-Rank Attention for Noisy Node Classification](https://mlsanity.com/paper/2402.09600/) (2024)
- [Rethinking Spectral Augmentation for Contrast-based Graph Self-Supervised Learning](https://mlsanity.com/paper/2405.19600/) (2024)

## Abstract (arXiv)

Although augmentations (e.g., perturbation of graph edges, image crops) boost the efficiency of Contrastive Learning (CL), feature level augmentation is another plausible, complementary yet not well researched strategy. Thus, we present a novel spectral feature argumentation for contrastive learning on graphs (and images). To this end, for each data view, we estimate a low-rank approximation per feature map and subtract that approximation from the map to obtain its complement. This is achieved by the proposed herein incomplete power iteration, a non-standard power iteration regime which enjoys two valuable byproducts (under mere one or two iterations): (i) it partially balances spectrum of the feature map, and (ii) it injects the noise into rebalanced singular values of the feature map (spectral augmentation). For two views, we align these rebalanced feature maps as such an improved alignment step can focus more on less dominant singular values of matrices of both views, whereas the spectral augmentation does not affect the spectral angle alignment (singular vectors are not perturbed). We derive the analytical form for: (i) the incomplete power iteration to capture its spectrum-balancing effect, and (ii) the variance of singular values augmented implicitly by the noise. We also show that the spectral augmentation improves the generalization bound. Experiments on graph/image datasets show that our spectral feature augmentation outperforms baselines, and is complementary with other augmentation strategies and compatible with various contrastive losses.