mlsanityData to 7 Oct 2026

SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption

arXiv 2106.15147 ↗First posted 29 Jun 2021cs.LG cs.AIcited by 151

Not on the map. Why?

Abstract (arXiv)

Self-supervised contrastive representation learning has proved incredibly successful in the vision and natural language domains, enabling state-of-the-art performance with orders of magnitude less labeled data. However, such methods are domain-specific and little has been done to leverage this technique on real-world tabular datasets. We propose SCARF, a simple, widely-applicable technique for contrastive learning, where views are formed by corrupting a random subset of features. When applied to pre-train deep neural networks on the 69 real-world, tabular classification datasets from the OpenML-CC18 benchmark, SCARF not only improves classification accuracy in the fully-supervised setting but does so also in the presence of label noise and in the semi-supervised setting where only a fraction of the available training data is labeled. We show that SCARF complements existing strategies and outperforms alternatives like autoencoders. We conduct comprehensive ablations, detailing the importance of a range of factors.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Builds on 75 of its 96 references that are in the map. First posted on arXiv 29 Jun 2021.

Builds on

75 of 96 references.

References · 75

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  5. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  7. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  9. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  13. 2016Colorful Image Colorizationcited by 1,113
  14. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  15. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  16. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  17. 2019When Does Label Smoothing Help?cited by 985
  18. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  19. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  20. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  21. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  22. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  23. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  24. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  25. 2020Contrastive Multi-View Representation Learning on Graphscited by 651
  26. 2014OpenML: networked science in machine learningcited by 523
  27. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  28. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  29. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
  30. 2016Adversarial Feature Learningcited by 350
  31. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  32. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  33. 2016Learning Representations for Automatic Colorizationcited by 251
  34. 2019Fast AutoAugmentcited by 243
  35. 2017Unsupervised Representation Learning by Sorting Sequencescited by 219
  36. 2016Joint Unsupervised Learning of Deep Representations and Image Clusterscited by 205
  37. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  38. 2020Does label smoothing mitigate label noise?cited by 204

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 12; 2023: 36; 2024: 40; 2025: 34; 2026: 28.

151 papers.

Sort:

By area:

  1. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  2. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  3. 2023A Cookbook of Self-Supervised Learningcited by 248
  4. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  5. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 105
  6. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  7. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  8. 2024Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learningcited by 79
  9. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 73
  10. 2023Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Datacited by 68
  11. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  12. 2022Transfer Learning with Deep Tabular Modelscited by 52
38 more
  1. 2023STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tablescited by 48
  2. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  3. 2024A Comprehensive Survey on Data Augmentationcited by 38
  4. 2024SwitchTab: Switched Autoencoders Are Effective Tabular Learnerscited by 37
  5. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  6. 2023TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specializationcited by 32
  7. 2024TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Datacited by 32
  8. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 29
  9. 2024Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Latercited by 28
  10. 2024PTaRL: Prototype-based Tabular Representation Learning via Space Calibrationcited by 27
  11. 2023Fascinating Supervisory Signals and Where to Find Them: Deep Anomaly Detection with Scale Learningcited by 25
  12. 2023Novel Class Discovery: an Introduction and Key Conceptscited by 24
  13. 2023MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinementcited by 24
  14. 2023ReConTab: Regularized Contrastive Representation Learning for Tabular Datacited by 24
  15. 2023Spatiotemporal Self-supervised Learning for Point Clouds in the Wildcited by 23
  16. 2023From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Modelscited by 22
  17. 2023Towards Cross-Table Masked Pretraining for Web Data Miningcited by 21
  18. 2024A Survey on Deep Tabular Learningcited by 18
  19. 2022A Hybrid Self-Supervised Learning Framework for Vertical Federated Learningcited by 17
  20. 2023Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Seriescited by 16
  21. 2024A Survey on Self-Supervised Learning for Non-Sequential Tabular Datacited by 15
  22. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 15
  23. 2023TabRet: Pre-training Transformer-based Tabular Models for Unseen Columnscited by 14
  24. 2023Self-supervised Representation Learning From Random Data Projectorscited by 14
  25. 2024MambaTab: A Plug-and-Play Model for Learning Tabular Datacited by 14
  26. 2024Graph Neural Networks for Tabular Data Learning: A Survey with Taxonomy and Directionscited by 13
  27. 2024CTSyn: A Foundation Model for Cross Tabular Data Generationcited by 13
  28. 2024T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Datacited by 13
  29. 2025On Finetuning Tabular Foundation Modelscited by 13
  30. 2022Don't Throw it Away! The Utility of Unlabeled Data in Fair Decision Makingcited by 12
  31. 2024Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domainscited by 12
  32. 2025STiL: Semi-supervised Tabular-Image Learning for Comprehensive Task-Relevant Information Exploration in Multimodal Classificationcited by 12
  33. 2022Is margin all you need? An extensive empirical study of active learning on tabular datacited by 10
  34. 2023Rethinking Data Augmentation for Tabular Data in Deep Learningcited by 10
  35. 2023MAP: A Model-agnostic Pretraining Framework for Click-through Rate Predictioncited by 10
  36. 2023TimeDRL: Disentangled Representation Learning for Multivariate Time-Seriescited by 10
  37. 2025TIME: TabPFN-Integrated Multimodal Engine for Robust Tabular-Image Learningcited by 10
  38. 2022TractoSCR: A Novel Supervised Contrastive Regression Framework for Prediction of Neurocognitive Measures Using Multi-Site Harmonized Diffusion MRI Tractographycited by 9

50 most cited of 151 All 151 (JSON)