mlsanityData to 7 Oct 2026

Few-Shot and Continual Learning with Attentive Independent Mechanisms

arXiv 2107.14053 ↗First posted 29 Jul 2021cs.LG cs.AI cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (DNNs) are known to perform well when deployed to test distributions that shares high similarity with the training distribution. Feeding DNNs with new data sequentially that were unseen in the training distribution has two major challenges -- fast adaptation to new tasks and catastrophic forgetting of old tasks. Such difficulties paved way for the on-going research on few-shot learning and continual learning. To tackle these problems, we introduce Attentive Independent Mechanisms (AIM). We incorporate the idea of learning using fast and slow weights in conjunction with the decoupling of the feature extraction and higher-order conceptual learning of a DNN. AIM is designed for higher-order conceptual learning, modeled by a mixture of experts that compete to learn independent concepts to solve a new task. AIM is a modular component that can be inserted into existing deep learning frameworks. We demonstrate its capability for few-shot learning by adding it to SIB and trained on MiniImageNet and CIFAR-FS, showing significant improvement. AIM is also applied to ANML and OML trained on Omniglot, CIFAR-100 and MiniImageNet to demonstrate its capability in continual learning. Code made publicly available at https://github.com/huang50213/AIM-Fewshot-Continual.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 29 Jul 2021.

Builds on

35 of 55 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2014Going Deeper with Convolutionscited by 4,231
  6. 2016Wide Residual Networkscited by 3,057
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2016Matching Networks for One Shot Learningcited by 2,246
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  10. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  11. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  12. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
23 more
  1. 2017Progressive Neural Architecture Searchcited by 483
  2. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  3. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  4. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  5. 2018Meta-learning with differentiable closed-form solverscited by 395
  6. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  7. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  8. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  9. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  10. 2017Meta Networkscited by 268
  11. 2019Recurrent Independent Mechanismscited by 219
  12. 2019Cross Attention Network for Few-shot Classificationcited by 218
  13. 2018Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learningcited by 208
  14. 2019Meta-Learning Representations for Continual Learningcited by 189
  15. 2019Boosting Few-Shot Visual Learning with Self-Supervisioncited by 140
  16. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 135
  17. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 129
  18. 2017Few-Shot Image Recognition by Predicting Parameters from Activationscited by 122
  19. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  20. 2019Finding Task-Relevant Features for Few-Shot Learning by Category Traversalcited by 101
  21. 2020Learning to Continually Learncited by 82
  22. 2020Empirical Bayes Transductive Meta-Learning with Synthetic Gradientscited by 48
  23. 2020NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networkscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 5; 2024: 1; 2025: 2; 2026: 1.

13 papers.

Sort:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2023S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learningcited by 23
  3. 2022On-device Training: A First Overview on Existing Systemscited by 18
  4. 2021Disentangled Feature Representation for Few-shot Image Classificationcited by 12
  5. 2023LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platformscited by 11
  6. 2022Adaptive Dimension Reduction and Variational Inference for Transductive Few-Shot Classificationcited by 10
  7. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  8. 2023Continual HyperTransformer: A Meta-Learner for Continual Few-Shot Learningcited by 3
  9. 2023Neural Attention Memorycited by 1
  10. 2022Expanding continual few-shot learning benchmarks to include recognition of specific instancescited by 0
  11. 2025MetaCLBench: Meta Continual Learning Benchmark on Resource-Constrained Edge Devicescited by 0
  12. 2025Hierarchical Mask-Enhanced Dual Reconstruction Network for Few-Shot Fine-Grained Image Classificationcited by 0
1 more
  1. 2026Are We Really Doing Few-Shot Learning? A Critical Examination of Pre-Training Assumptionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhanced Continual Learning of Vision-Language Models with Model Fusion
  2. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
  3. 2024Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning
  4. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
  5. 2025Modular Embedding Recomposition for Incremental Learning
  6. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models
  7. 2026SAMBAR: Selective Anchoring via Method of Multipliers for Balanced Knowledge Acquisition and Retention in Vision-Language-Action Models
  8. 2026Cortex-Inspired Continual Learning: Unsupervised Instantiation and Recovery of Functional Task Networks