# Disentangling the Effects of Data Augmentation and Format Transform in Self-Supervised Learning of Image Representations

arXiv: https://arxiv.org/abs/2312.02205 · HTML: https://mlsanity.com/paper/2312.02205/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 2 Dec 2023.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [A Fourier-based Framework for Domain Generalization](https://mlsanity.com/paper/2105.11120/) (2021)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases](https://mlsanity.com/paper/2007.13916/) (2020)
- [Improving Transferability of Representations via Augmentation-Aware Self-Supervision](https://mlsanity.com/paper/2111.09613/) (2021)
- [Frequency Domain Image Translation: More Photo-realistic, Better Identity-preserving](https://mlsanity.com/paper/2011.13611/) (2020)
- [Federated Self-Supervised Learning of Multi-Sensor Representations for Embedded Intelligence](https://mlsanity.com/paper/2007.13018/) (2020)
- [Multi-Format Contrastive Learning of Audio Representations](https://mlsanity.com/paper/2103.06508/) (2021)

## Built on by (most cited first, 2 of 2)

- [Augmentations vs Algorithms: What Works in Self-Supervised Learning](https://mlsanity.com/paper/2403.05726/) (2024)
- [Self-Bootstrapping for Versatile Test-Time Adaptation](https://mlsanity.com/paper/2504.08010/) (2025)

## Abstract (arXiv)

Self-Supervised Learning (SSL) enables training performant models using limited labeled data. One of the pillars underlying vision SSL is the use of data augmentations/perturbations of the input which do not significantly alter its semantic content. For audio and other temporal signals, augmentations are commonly used alongside format transforms such as Fourier transforms or wavelet transforms. Unlike augmentations, format transforms do not change the information contained in the data; rather, they express the same information in different coordinates. In this paper, we study the effects of format transforms and augmentations both separately and together on vision SSL. We define augmentations in frequency space called Fourier Domain Augmentations (FDA) and show that training SSL models on a combination of these and image augmentations can improve the downstream classification accuracy by up to 1.3% on ImageNet-1K. We also show improvements against SSL baselines in few-shot and transfer learning setups using FDA. Surprisingly, we also observe that format transforms can improve the quality of learned representations even without augmentations; however, the combination of the two techniques yields better quality.