# COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning

arXiv: https://arxiv.org/abs/2206.04726 · HTML: https://mlsanity.com/paper/2206.04726/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 52 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 30 of 37)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [Graph Contrastive Learning with Adaptive Augmentation](https://mlsanity.com/paper/2010.14945/) (2020)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Self-supervised Graph Learning for Recommendation](https://mlsanity.com/paper/2010.10783/) (2020)
- [Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks](https://mlsanity.com/paper/2007.02901/) (2020)
- [Graph Representation Learning via Graphical Mutual Information Maximization](https://mlsanity.com/paper/2002.01169/) (2020)
- [Adversarial Graph Augmentation to Improve Graph Contrastive Learning](https://mlsanity.com/paper/2106.05819/) (2021)
- [Are Graph Augmentations Necessary? Simple Graph Contrastive Learning for Recommendation](https://mlsanity.com/paper/2112.08679/) (2021)
- [Dataset Augmentation in Feature Space](https://mlsanity.com/paper/1702.05538/) (2017)
- [Implicit Semantic Data Augmentation for Deep Networks](https://mlsanity.com/paper/1909.12220/) (2019)
- [Graph Barlow Twins: A self-supervised representation learning framework for graphs](https://mlsanity.com/paper/2106.02466/) (2021)
- [Graph-based Semi-supervised Learning: A Comprehensive Review](https://mlsanity.com/paper/2102.13303/) (2021)
- [Modeling Scale-free Graphs with Hyperbolic Geometry for Knowledge-aware Recommendation](https://mlsanity.com/paper/2108.06468/) (2021)
- [HRCF: Enhancing Collaborative Filtering via Hyperbolic Geometric Regularization](https://mlsanity.com/paper/2204.08176/) (2022)
- [Power Normalizations in Fine-grained Image, Few-shot Image and Graph Classification](https://mlsanity.com/paper/2012.13975/) (2020)

## Built on by (most cited first, 30 of 39)

- [Spectral Feature Augmentation for Graph Contrastive Learning and Beyond](https://mlsanity.com/paper/2212.01026/) (2022)
- [3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition](https://mlsanity.com/paper/2303.14474/) (2023)
- [GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning](https://mlsanity.com/paper/2310.07365/) (2023)
- [A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy](https://mlsanity.com/paper/2302.10637/) (2023)
- [Uncertainty-DTW for Time Series and Sequences](https://mlsanity.com/paper/2211.00005/) (2022)
- [GraphLoRA: Structure-Aware Contrastive Low-Rank Adaptation for Cross-Graph Transfer Learning](https://mlsanity.com/paper/2409.16670/) (2024)
- [HomoGCL: Rethinking Homophily in Graph Contrastive Learning](https://mlsanity.com/paper/2306.09614/) (2023)
- [Graph Soft-Contrastive Learning via Neighborhood Ranking](https://mlsanity.com/paper/2209.13964/) (2022)
- [kHGCN: Tree-likeness Modeling via Continuous and Discrete Curvature Learning](https://mlsanity.com/paper/2212.01793/) (2022)
- [Wiener Graph Deconvolutional Network Improves Graph Self-Supervised Learning](https://mlsanity.com/paper/2206.12933/) (2022)
- [Traffic Forecasting on New Roads Using Spatial Contrastive Pre-Training (SCPT)](https://mlsanity.com/paper/2305.05237/) (2023)
- [Masked Graph Autoencoder with Non-discrete Bandwidths](https://mlsanity.com/paper/2402.03814/) (2024)
- [Noise-Resilient Unsupervised Graph Representation Learning via Multi-Hop Feature Quality Estimation](https://mlsanity.com/paper/2407.19944/) (2024)
- [Rethinking and Simplifying Bootstrapped Graph Latents](https://mlsanity.com/paper/2312.02619/) (2023)
- [Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning](https://mlsanity.com/paper/2310.18209/) (2023)
- [A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective](https://mlsanity.com/paper/2403.16137/) (2024)
- [Towards Generalizable Graph Contrastive Learning: An Information Theory Perspective](https://mlsanity.com/paper/2211.10929/) (2022)
- [MARIO: Model Agnostic Recipe for Improving OOD Generalization of Graph Contrastive Learning](https://mlsanity.com/paper/2307.13055/) (2023)
- [Graph Component Contrastive Learning for Concept Relatedness Estimation](https://mlsanity.com/paper/2206.12556/) (2022)
- [Bootstrap Latents of Nodes and Neighbors for Graph Self-Supervised Learning](https://mlsanity.com/paper/2408.05087/) (2024)
- [Negative-Free Self-Supervised Gaussian Embedding of Graphs](https://mlsanity.com/paper/2411.01157/) (2024)
- [Spectral-Aware Augmentation for Enhanced Graph Representation Learning](https://mlsanity.com/paper/2310.13845/) (2023)
- [From Overfitting to Robustness: Quantity, Quality, and Variety Oriented Negative Sample Selection in Graph Contrastive Learning](https://mlsanity.com/paper/2406.15044/) (2024)
- [Overcoming Pitfalls in Graph Contrastive Learning Evaluation: Toward Comprehensive Benchmarks](https://mlsanity.com/paper/2402.15680/) (2024)
- [InfoNCE is a Free Lunch for Semantically guided Graph Contrastive Learning](https://mlsanity.com/paper/2505.06282/) (2025)
- [Momentum Contrastive Pre-training for Question Answering](https://mlsanity.com/paper/2212.05762/) (2022)
- [Negative as Positive: Enhancing Out-of-distribution Generalization for Graph Contrastive Learning](https://mlsanity.com/paper/2405.16224/) (2024)
- [FedHGN: A Federated Framework for Heterogeneous Graph Neural Networks](https://mlsanity.com/paper/2305.09729/) (2023)
- [Hi-GMAE: Hierarchical Graph Masked Autoencoders](https://mlsanity.com/paper/2405.10642/) (2024)
- [Topology Reorganized Graph Contrastive Learning with Mitigating Semantic Drift](https://mlsanity.com/paper/2407.16726/) (2024)

## Abstract (arXiv)

Graph contrastive learning (GCL) improves graph representation learning, leading to SOTA on various downstream tasks. The graph augmentation step is a vital but scarcely studied step of GCL. In this paper, we show that the node embedding obtained via the graph augmentations is highly biased, somewhat limiting contrastive models from learning discriminative features for downstream tasks. Thus, instead of investigating graph augmentation in the input space, we alternatively propose to perform augmentations on the hidden features (feature augmentation). Inspired by so-called matrix sketching, we propose COSTA, a novel COvariance-preServing feaTure space Augmentation framework for GCL, which generates augmented features by maintaining a "good sketch" of original features. To highlight the superiority of feature augmentation with COSTA, we investigate a single-view setting (in addition to multi-view one) which conserves memory and computations. We show that the feature augmentation with COSTA achieves comparable/better results than graph augmentation based models.