mlsanityData to 14 Oct 2026

Learning to Learn with Contrastive Meta-Objective

arXiv 2410.05975 ↗First posted 8 Oct 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Meta-learning enables learning systems to adapt quickly to new tasks, similar to humans. Different meta-learning approaches all work under/with the mini-batch episodic training framework. Such framework naturally gives the information about task identity, which can serve as additional supervision for meta-training to improve generalizability. We propose to exploit task identity as additional supervision in meta-training, inspired by the alignment and discrimination ability which is is intrinsic in human's fast learning. This is achieved by contrasting what meta-learners learn, i.e., model representations. The proposed ConML is evaluating and optimizing the contrastive meta-objective under a problem- and learner-agnostic meta-training framework. We demonstrate that ConML integrates seamlessly with existing meta-learners, as well as in-context learning models, and brings significant boost in performance with small implementation cost.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 53 of its 76 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

53 of 76 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  8. 2020Supervised Contrastive Learningcited by 2,829
  9. 2016Matching Networks for One Shot Learningcited by 2,255
  10. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  12. 2019Contrastive Multiview Codingcited by 1,301
38 more
  1. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  3. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,143
  4. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  5. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  6. 2022Transformers learn in-context by gradient descentcited by 732
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  9. 2016Learning to learn by gradient descent by gradient descentcited by 620
  10. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  11. 2021MetaICL: Learning to Learn In Contextcited by 454
  12. 2018Conditional Neural Processescited by 426
  13. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 315
  14. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  15. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  16. 2015The Benefit of Multitask Representation Learningcited by 195
  17. 2020CrossTransformers: spatially-aware few-shot transfercited by 174
  18. 2019Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processescited by 131
  19. 2022Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Differencecited by 126
  20. 2019Improved Few-Shot Visual Classificationcited by 115
  21. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  22. 2016Low Data Drug Discovery with One-shot Learningcited by 91
  23. 2021Few-Shot Graph Learning for Molecular Property Predictioncited by 81
  24. 2019Meta-Learning of Neural Architectures for Few-Shot Learningcited by 73
  25. 2021Property-Aware Relation Networks for Few-Shot Molecular Property Predictioncited by 41
  26. 2022Rethinking Generalization in Few-Shot Classificationcited by 36
  27. 2020Adaptive Task Sampling for Meta-Learningcited by 29
  28. 2023Context-enriched molecule representations improve few-shot drug discoverycited by 24
  29. 2022Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property Predictioncited by 21
  30. 2023Context-Aware Meta-Learningcited by 21
  31. 2018Meta Architecture Searchcited by 19
  32. 2020Function Contrastive Learning of Transferable Meta-Representationscited by 19
  33. 2021Few-Shot Learning with a Strong Teachercited by 16
  34. 2021A contrastive rule for meta-learningcited by 14
  35. 2021Memory Efficient Meta-Learning with Large Imagescited by 14
  36. 2024Robust Fast Adaptation from Adversarially Explicit Task Distribution Generationcited by 13
  37. 2021On Contrastive Representations of Stochastic Processescited by 12
  38. 2023Towards Task Sampler Learning for Meta-Learningcited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025DRESS: Disentangled Representation-based Self-Supervised Meta-Learning for Diverse Taskscited by 2
  2. 2026Compositional Meta-Learning for Mitigating Task Heterogeneity in Physics-Informed Neural Networkscited by 1
  3. 2026Provable Data Scaling Law for Meta Learning via Complexity Minimizationcited by 0