mlsanityData to 7 Oct 2026

Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning

arXiv 2201.03529 ↗First posted 10 Jan 2022cs.LG cs.CVcited by 83

Not on the map. Why?

Abstract (arXiv)

Transfer-learning methods aim to improve performance in a data-scarce target domain using a model pretrained on a data-rich source domain. A cost-efficient strategy, linear probing, involves freezing the source model and training a new classification head for the target domain. This strategy is outperformed by a more costly but state-of-the-art method -- fine-tuning all parameters of the source model to the target domain -- possibly because fine-tuning allows the model to leverage useful information from intermediate layers which is otherwise discarded by the later pretrained layers. We explore the hypothesis that these intermediate layers might be directly exploited. We propose a method, Head-to-Toe probing (Head2Toe), that selects features from all layers of the source model to train a classification head for the target-domain. In evaluations on the VTAB-1k, Head2Toe matches performance obtained with fine-tuning on average while reducing training and storage cost hundred folds or more, but critically, for out-of-distribution transfer, Head2Toe outperforms fine-tuning.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 46 of its 91 references that are in the map. First posted on arXiv 10 Jan 2022.

Builds on

46 of 91 references.

References · 46

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  8. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  10. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  11. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  12. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
34 more
  1. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  2. 2018Do Better ImageNet Models Transfer Better?cited by 560
  3. 2017Learning multiple visual domains with residual adapterscited by 553
  4. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  5. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  6. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  7. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  8. 2021Projected GANs Converge Fastercited by 204
  9. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 166
  10. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  11. 2014Hypercolumns for Object Segmentation and Fine-grained Localizationcited by 128
  12. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  13. 2019Concrete Autoencoders for Differentiable Feature Selection and Reconstructioncited by 90
  14. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  15. 2015Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networkscited by 80
  16. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  17. 2020A Universal Representation Transformer Layer for Few-Shot Image Classificationcited by 65
  18. 2021Learning a Universal Template for Few-shot Dataset Generalizationcited by 61
  19. 2021No One Representation to Rule Them All: Overlapping Features of Training Methodscited by 57
  20. 2021Universal Representation Learning from Multiple Domains for Few-shot Classificationcited by 56
  21. 2020Scalable Transfer Learning with Expert Modelscited by 48
  22. 2017Universal representations:The missing link between faces, text, planktons, and cat breedscited by 45
  23. 2017Transfer learning for music classification and regression taskscited by 36
  24. 2020A Survey on Transfer Learning in Natural Language Processingcited by 33
  25. 2020Cross-Domain Few-Shot Learning by Representation Fusioncited by 32
  26. 2021Fast Adaptation with Linearized Neural Networkscited by 32
  27. 2021Universal Paralinguistic Speech Representations Using Self-Supervised Conformerscited by 32
  28. 2020Gradients as Features for Deep Representation Learningcited by 30
  29. 2020Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoderscited by 27
  30. 2020Deep Ensembles for Low-Data Transfer Learningcited by 19
  31. 2017Forward-Backward Selection with Early Droppingcited by 18
  32. 2021Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmarkcited by 16
  33. 2017Multi-Level and Multi-Scale Feature Aggregation Using Pre-trained Convolutional Neural Networks for Music Auto-taggingcited by 8
  34. 2020Deep Transfer Learning with Ridge Regressioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 26; 2024: 22; 2025: 13; 2026: 9.

83 papers.

Sort:

By area:

  1. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  2. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  3. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  4. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  5. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  6. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  7. 2022Few-shot Image Generation via Adaptation-Aware Kernel Modulationcited by 33
  8. 2022Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learningcited by 25
  9. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  10. 2022No Reason for No Supervision: Improved Generalization in Supervised Modelscited by 23
  11. 2023A Closer Look at Model Adaptation using Feature Distortion and Simplicity Biascited by 20
  12. 2023Exploring Incompatible Knowledge Transfer in Few-shot Image Generationcited by 19
38 more
  1. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  2. 2023BIRB: A Generalization Benchmark for Information Retrieval in Bioacousticscited by 19
  3. 2025BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Modelscited by 18
  4. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  5. 2023Exploring Efficient Few-shot Adaptation for Vision Transformerscited by 16
  6. 2023Objectives Matter: Understanding the Impact of Self-Supervised Objectives on Vision Transformer Representationscited by 15
  7. 2023Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptationcited by 14
  8. 2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
  9. 2025Intermediate Layer Classifiers for OOD generalizationcited by 14
  10. 2023Less is More: Selective Layer Finetuning with SubTuningcited by 13
  11. 2023Guiding The Last Layer in Federated Learning with Pre-Trained Modelscited by 13
  12. 2024What Variables Affect Out-of-Distribution Generalization in Pretrained Models?cited by 13
  13. 2024ViTally Consistent: Scaling Biological Representation Learning for Cell Microscopycited by 13
  14. 2023Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learningcited by 11
  15. 2024SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learningcited by 11
  16. 2024MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mambacited by 11
  17. 2025PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialoguecited by 8
  18. 2022Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based modelscited by 7
  19. 2023Visual DNA: Representing and Comparing Images using Distributions of Neuron Activationscited by 7
  20. 2023Lightweight Learner for Shared Knowledge Lifelong Learningcited by 7
  21. 2023Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Datacited by 7
  22. 2023CLR: Channel-wise Lightweight Reprogramming for Continual Learningcited by 6
  23. 2024Effective Layer Pruning Through Similarity Metric Perspectivecited by 6
  24. 2024MapSAM: Adapting Segment Anything Model for Automated Feature Detection in Historical Mapscited by 6
  25. 2023Substance or Style: What Does Your Image Embedding Know?cited by 5
  26. 2024Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcitycited by 5
  27. 2024Data Selection for Transfer Unlearningcited by 5
  28. 2025Leveraging Intermediate Representations for Better Out-of-Distribution Detectioncited by 5
  29. 2022Understanding the Robustness of Multi-Exit Models under Common Corruptionscited by 4
  30. 2024LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Viewscited by 4
  31. 2024Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approachcited by 4
  32. 2026Attentive multilayer fusion for vision transformerscited by 4
  33. 2023What Happens During Finetuning of Vision Transformers: An Invariance Based Investigationcited by 3
  34. 2023Dynamic Visual Prompt Tuning for Parameter Efficient Transfer Learningcited by 3
  35. 2024AutoFT: Learning an Objective for Robust Fine-Tuningcited by 3
  36. 2024Fine-tuning with Very Large Dropoutcited by 3
  37. 2024Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweightingcited by 3
  38. 2025Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalizationcited by 3

50 most cited of 83 All 83 (JSON)