HOP to the Next Tasks and Domains for Continual Learning in NLP
Not on the map. Why?
Abstract (arXiv)
Continual Learning (CL) aims to learn a sequence of problems (i.e., tasks and domains) by transferring knowledge acquired on previous problems, whilst avoiding forgetting of past ones. Different from previous approaches which focused on CL for one NLP task or domain in a specific use-case, in this paper, we address a more general CL setting to learn from a sequence of problems in a unique framework. Our method, HOP, permits to hop across tasks and domains by addressing the CL problem along three directions: (i) we employ a set of adapters to generalize a large pre-trained model to unseen problems, (ii) we compute high-order moments over the distribution of embedded representations to distinguish independent and correlated statistics across different tasks and domains, (iii) we process this enriched information with auxiliary heads specialized for each end problem. Extensive experimental campaign on 4 NLP applications, 5 benchmarks and 2 CL setups demonstrates the effectiveness of our HOP.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 48 of its 87 references that are in the map. First posted on arXiv 28 Feb 2024.
Builds on
48 of 87 references.
References · 48
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2016Learning without Forgettingcited by 2,465
- 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
- 2017Gradient Episodic Memory for Continual Learningcited by 1,871
- 2016Progressive Neural Networkscited by 1,332
- 2017Continual Learning with Deep Generative Replaycited by 1,152
- 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
- 2018Efficient Lifelong Learning with A-GEMcited by 1,044
- 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
36 more
- 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
- 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
- 2019Three scenarios for continual learningcited by 610
- 2017Variational Continual Learningcited by 335
- 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
- 2018Selective Experience Replay for Lifelong Learningcited by 239
- 2018Reinforced Continual Learningcited by 186
- 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
- 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
- 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
- 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
- 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 144
- 2019Incremental Learning Techniques for Semantic Segmentationcited by 141
- 2016Less-forgetting Learning in Deep Neural Networkscited by 97
- 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
- 2021RECALL: Replay-based Continual Learning in Semantic Segmentationcited by 83
- 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
- 2020Continual Learning in Task-Oriented Dialogue Systemscited by 82
- 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 80
- 2017Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalizationcited by 65
- 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
- 2022Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interferencecited by 44
- 2019Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answeringcited by 33
- 2020Lifelong Language Knowledge Distillationcited by 31
- 2021CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Taskscited by 29
- 2021Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translationcited by 28
- 2019Continual Learning for Sentence Representations Using Conceptorscited by 22
- 2018Lifelong Learning for Sentiment Classificationcited by 21
- 2018Progressive Memory Banks for Incremental Domain Adaptationcited by 12
- 2021Continual Learning with Knowledge Transfer for Sentiment Classificationcited by 12
- 2019Forward and Backward Knowledge Transfer for Sentiment Classificationcited by 9
- 2020Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learningcited by 9
- 2023Online Continual Learning for Robust Indoor Object Recognitioncited by 9
- 2023Online Continual Learning in Keyword Spotting for Low-Resource Devices via Pooling High-Order Temporal Statisticscited by 9
- 2021A Student-Teacher Architecture for Dialog Domain Adaptation under the Meta-Learning Settingcited by 5
- 2023Continual Graph Convolutional Network for Text Classificationcited by 4
Built on by
Citing papers per year: 2024: 2; 2025: 0; 2026: 1.
3 papers.