mlsanityData to 7 Oct 2026

Self-supervised Representation Learning From Random Data Projectors

arXiv 2310.07756 ↗First posted 11 Oct 2023cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Self-supervised representation learning~(SSRL) has advanced considerably by exploiting the transformation invariance assumption under artificially designed data augmentations. While augmentation-based SSRL algorithms push the boundaries of performance in computer vision and natural language processing, they are often not directly applicable to other data modalities, and can conflict with application-specific data augmentation constraints. This paper presents an SSRL approach that can be applied to any data modality and network architecture because it does not rely on augmentations or masking. Specifically, we show that high-quality data representations can be learned by reconstructing random data projections. We evaluate the proposed approach on a wide range of representation learning tasks that span diverse modalities and real-world applications. We show that it outperforms multiple state-of-the-art SSRL baselines. Due to its wide applicability and strong empirical results, we argue that learning from randomness is a fruitful research direction worthy of attention and further study.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 47 of its 71 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

47 of 71 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  12. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
35 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  3. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  4. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  5. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  6. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  7. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  8. 2016Time Series Classification from Scratch with Deep Neural Networks: A Strong Baselinecited by 358
  9. 2021Time-Series Representation Learning via Temporal and Contextual Contrastingcited by 342
  10. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  11. 2017Multitask learning and benchmarking with clinical time series datacited by 287
  12. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  13. 2023A Cookbook of Self-Supervised Learningcited by 248
  14. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  15. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  16. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  17. 2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 148
  18. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  19. 2017Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networkscited by 130
  20. 2020An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networkscited by 128
  21. 2020Flows for simultaneous manifold learning and density estimationcited by 117
  22. 2018Conditional BERT Contextual Augmentationcited by 107
  23. 2021SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learningcited by 101
  24. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  25. 2020Subject-Aware Contrastive Learning for Biosignalscited by 75
  26. 2020Towards Domain-Agnostic Contrastive Learningcited by 75
  27. 2021Decentralized Federated Learning through Proxy Model Sharingcited by 56
  28. 2023Time Series Contrastive Learning with Information-Aware Augmentationscited by 52
  29. 2023TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoderscited by 42
  30. 2022Rethinking the Augmentation Module in Contrastive Learning: Learning Hierarchical Augmentation Invariance with Expanded Viewscited by 37
  31. 2016Parameterized Machine Learning for High-Energy Physicscited by 33
  32. 2021Neighborhood Contrastive Learning Applied to Online Patient Monitoringcited by 32
  33. 2022RandStainNA: Learning Stain-Agnostic Features from Histology Slides by Bridging Stain Augmentation and Normalizationcited by 32
  34. 2021Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Taskscited by 15
  35. 2023Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Headcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 5; 2026: 4.

14 papers.

Sort:

  1. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  2. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  3. 2024A Survey on Self-Supervised Learning for Non-Sequential Tabular Datacited by 15
  4. 2024T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Datacited by 13
  5. 2023Data-Efficient Multimodal Fusion on a Single GPUcited by 11
  6. 2026Causal Foundation Models with Continuous Treatmentscited by 4
  7. 2024Towards Active Participant Centric Vertical Federated Learning: Some Representations May Be All You Needcited by 2
  8. 2025DRESS: Disentangled Representation-based Self-Supervised Meta-Learning for Diverse Taskscited by 2
  9. 2025Self-Supervised Representation Learning as Mutual Information Maximizationcited by 0
  10. 2025Beyond Augmentation: Leveraging Inter-Instance Relation in Self-Supervised Representation Learningcited by 0
  11. 2025Learning Without Augmenting: Unsupervised Time Series Representation Learning via Frame Projectionscited by 0
  12. 2026Training Fair Tabular Foundation Modelscited by 0
2 more
  1. 2026Unifying Conformal Language Tasks with In-Context Ensemblescited by 0
  2. 2026Domain Generalization for Smartphone-Based Human Activity Recognition: A Systematic Analysis of Components and Interactionscited by 0