Self-Masking Networks for Unsupervised Adaptation
Not on the map. Why?
Abstract (arXiv)
With the advent of billion-parameter foundation models, efficient fine-tuning has become increasingly important for the adaptation of models to downstream tasks. However, especially in computer vision, it can be hard to achieve good performance when access to quality labeled data is lacking. In this work, we propose a method adapting pretrained generalist models in a self-supervised manner by learning binary masks. These self-supervised masking networks (SMNs) are up to 79x more efficient to store and significantly improve performance on label-efficient downstream tasks. We validate the usefulness of learning binary masks as a fine-tuning method on 8 datasets and 3 model architectures, and we demonstrate the effectiveness of SMNs in 3 label-efficient settings.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 11 Sep 2024.
Builds on
40 of 57 references.
References · 40
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2022Visual Prompt Tuningcited by 1,674
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
- 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
28 more
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
- 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
- 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
- 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
- 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
- 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
- 2017Learning multiple visual domains with residual adapterscited by 553
- 2019Self-labelling via simultaneous clustering and representation learningcited by 525
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
- 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
- 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
- 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
- 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
- 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
- 2020Supermasks in Superpositioncited by 223
- 2020How Well Do Self-Supervised Models Transfer?cited by 213
- 2016Trained Ternary Quantizationcited by 203
- 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
- 2016Ternary Weight Networkscited by 195
- 2021Block Pruning For Faster Transformerscited by 176
- 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
- 2020Learning to Balance Specificity and Invariance for In and Out of Domain Generalizationcited by 128
- 2018Adversarial Reprogramming of Neural Networkscited by 115
- 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
- 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
- 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 81
- 2019Discovering Neural Wiringscited by 56
- 2022On Label Granularity and Object Localizationcited by 12
Built on by
Citing papers per year: 2025: 2.
2 papers.