mlsanityData to 14 Oct 2026

SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learning

arXiv 2310.02751 ↗First posted 4 Oct 2023cs.LG cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

In this paper, we hypothesize that gradient-based meta-learning (GBML) implicitly suppresses the Hessian along the optimization trajectory in the inner loop. Based on this hypothesis, we introduce an algorithm called SHOT (Suppressing the Hessian along the Optimization Trajectory) that minimizes the distance between the parameters of the target and reference models to suppress the Hessian in the inner loop. Despite dealing with high-order terms, SHOT does not increase the computational complexity of the baseline model much. It is agnostic to both the algorithm and architecture used in GBML, making it highly versatile and applicable to any GBML baseline. To validate the effectiveness of SHOT, we conduct empirical tests on standard few-shot learning tasks and qualitatively analyze its dynamics. We confirm our hypothesis empirically and demonstrate that SHOT outperforms the corresponding baseline. Code is available at: https://github.com/JunHoo-Lee/SHOT

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

29 of 40 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2016Matching Networks for One Shot Learningcited by 2,255
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
17 more
  1. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,143
  2. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  3. 2019Meta-Learning with Implicit Gradientscited by 525
  4. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  5. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  6. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
  7. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  8. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  9. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  10. 2019Meta-Learning with Warped Gradient Descentcited by 98
  11. 2020Meta-Learning with Adaptive Hyperparameterscited by 79
  12. 2019Meta-Curvaturecited by 70
  13. 2017On weight initialization in deep neural networkscited by 67
  14. 2019Torchmeta: A Meta-Learning library for PyTorchcited by 52
  15. 2021Meta-learning with negative learning ratescited by 11
  16. 2022On Enforcing Better Conditioned Meta-Learning for Rapid Few-Shot Adaptationcited by 7
  17. 2020Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Netscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Any-Way Meta Learningcited by 1