mlsanityData to 7 Oct 2026

No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations

arXiv 2407.10964 ↗First posted 15 Jul 2024cs.CV cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper introduces FUNGI, Features from UNsupervised GradIents, a method to enhance the features of transformer encoders by leveraging self-supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. These gradients are projected to a lower dimension and then concatenated with the model's output embedding. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio. Across backbones spanning various sizes and pretraining strategies, FUNGI features provide consistent performance improvements over the embeddings. We also show that using FUNGI features can benefit linear classification, clustering and image retrieval, and that they significantly improve the retrieval-based in-context scene understanding abilities of pretrained models, for example improving upon DINO by +17% for semantic segmentation - without any training.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 33 of its 96 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

33 of 96 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  10. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
21 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  3. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  4. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  6. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  7. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  8. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  9. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  10. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 338
  11. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  12. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  13. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  14. 2016Learning feed-forward one-shot learnerscited by 93
  15. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 82
  16. 2019Test-Time Training with Self-Supervision for Generalization under Distribution Shiftscited by 55
  17. 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
  18. 2023Towards In-context Scene Understandingcited by 45
  19. 2019Cyanure: An Open-Source Toolbox for Empirical Risk Minimization for Python, C++, and soon morecited by 16
  20. 2024Frozen Feature Augmentation for Few-Shot Image Classificationcited by 13
  21. 2019Accelerating Large-Scale Inference with Anisotropic Vector Quantizationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  2. 2025DIP: Unsupervised Dense In-Context Post-training of Visual Representationscited by 3
  3. 2025Benchmarking Image Embeddings for E-Commerce: Evaluating Off-the Shelf Foundation Models, Fine-Tuning Strategies and Practical Trade-offscited by 1
  4. 2026Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcarecited by 1
  5. 2026Is Domain Adaptation Always Helpful? A Frozen-Backbone Study of Cross-Domain Sentiment Transfercited by 1