mlsanityData to 7 Oct 2026

Recasting Continual Learning as Sequence Modeling

arXiv 2310.11952 ↗First posted 18 Oct 2023cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

In this work, we aim to establish a strong connection between two significant bodies of machine learning research: continual learning and sequence modeling. That is, we propose to formulate continual learning as a sequence modeling problem, allowing advanced sequence models to be utilized for continual learning. Under this formulation, the continual learning process becomes the forward pass of a sequence model. By adopting the meta-continual learning (MCL) framework, we can train the sequence model at the meta-level, on multiple continual learning episodes. As a specific example of our new formulation, we demonstrate the application of Transformers and their efficient variants as MCL methods. Our experiments on seven benchmarks, covering both classification and regression, show that sequence models can be an attractive solution for general MCL.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

29 of 46 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2016Progressive Neural Networkscited by 1,332
17 more
  1. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  2. 2020Efficient Transformers: A Surveycited by 784
  3. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  4. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  5. 2016MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognitioncited by 621
  6. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  7. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  8. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  9. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  10. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 251
  11. 2019Massively Multilingual Neural Machine Translationcited by 246
  12. 2019Meta-Learning Representations for Continual Learningcited by 189
  13. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  14. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  15. 2019Training Neural Networks with Local Error Signalscited by 135
  16. 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 133
  17. 2020Learning to Continually Learncited by 82

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 3; 2026: 2.

11 papers.

  1. 2023When Meta-Learning Meets Online and Continual Learning: A Surveycited by 12
  2. 2025Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planningcited by 12
  3. 2024Learning to Continually Learn with the Bayesian Principlecited by 8
  4. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 7
  5. 2023Metalearning Continual Learning Algorithmscited by 6
  6. 2024Transformers for Supervised Online Continual Learningcited by 5
  7. 2024Overcoming classic challenges for artificial neural networks by providing incentives and practicecited by 3
  8. 2024Learning Mamba as a Continual Learner: Meta-Learning Selective State Space Models for Continual Learningcited by 1
  9. 2025Meta-Learning Transformers to Improve In-Context Generalizationcited by 0
  10. 2026Finding Structure in Continual Learningcited by 0
  11. 2026Task Switching Without Forgetting via Proximal Decouplingcited by 0