mlsanityData to 14 Oct 2026

AASAE: Augmentation-Augmented Stochastic Autoencoders

arXiv 2107.12329 ↗First posted 26 Jul 2021cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Recent methods for self-supervised learning can be grouped into two paradigms: contrastive and non-contrastive approaches. Their success can largely be attributed to data augmentation pipelines which generate multiple views of a single input that preserve the underlying semantics. In this work, we introduce augmentation-augmented stochastic autoencoders (AASAE), yet another alternative to self-supervised learning, based on autoencoding. We derive AASAE starting from the conventional variational autoencoder (VAE), by replacing the KL divergence regularization, which is agnostic to the input domain, with data augmentations that explicitly encourage the internal representations to encode domain-specific invariances and equivariances. We empirically evaluate the proposed AASAE on image classification, similar to how recent contrastive and non-contrastive learning algorithms have been evaluated. Our experiments confirm the effectiveness of data augmentation as a replacement for KL divergence regularization. The AASAE outperforms the VAE by 30% on CIFAR-10, 40% on STL-10 and 45% on Imagenet. On CIFAR-10 and STL-10, the results for AASAE are largely comparable to the state-of-the-art algorithms for self-supervised learning.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 26 Jul 2021.

Builds on

46 of 63 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2013Auto-Encoding Variational Bayescited by 4,053
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
34 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  5. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  6. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  7. 2019Contrastive Multiview Codingcited by 1,301
  8. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  9. 2016Colorful Image Colorizationcited by 1,114
  10. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  12. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  13. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  14. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  15. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  16. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  17. 2020Contrastive Learning with Hard Negative Samplescited by 628
  18. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  19. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  20. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  21. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 472
  22. 2020Debiased Contrastive Learningcited by 356
  23. 2017Learning Generative Models with Sinkhorn Divergencescited by 314
  24. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  25. 2017Multi-task Self-Supervised Visual Learningcited by 200
  26. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 198
  27. 2020SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustnesscited by 96
  28. 2020BYOL works even without batch statisticscited by 95
  29. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  30. 2020Understanding Self-supervised Learning with Dual Deep Networkscited by 62
  31. 2020A Framework For Contrastive Self-Supervised Learning And Designing A New Approachcited by 52
  32. 2018Avoiding Latent Variable Collapse With Generative Skip Modelscited by 47
  33. 2015Denoising Criterion for Variational Auto-Encoding Frameworkcited by 43
  34. 2018Leveraging the Exact Likelihood of Deep Latent Variable Modelscited by 23

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023ResWCAE: Biometric Pattern Image Denoising Using Residual Wavelet-Conditioned Autoencodercited by 2