# RVAE-LAMOL: Residual Variational Autoencoder to Enhance Lifelong Language Learning

arXiv: https://arxiv.org/abs/2205.10857 · HTML: https://mlsanity.com/paper/2205.10857/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 22 May 2022.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [Episodic Memory in Lifelong Language Learning](https://mlsanity.com/paper/1906.01076/) (2019)
- [LAMOL: LAnguage MOdeling for Lifelong Language Learning](https://mlsanity.com/paper/1909.03329/) (2019)
- [Learning to Generate Reviews and Discovering Sentiment](https://mlsanity.com/paper/1704.01444/) (2017)
- [Invariant Rationalization](https://mlsanity.com/paper/2003.09772/) (2020)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior](https://mlsanity.com/paper/1908.07181/) (2019)
- [Efficient Meta Lifelong-Learning with Limited Memory](https://mlsanity.com/paper/2010.02500/) (2020)
- [Transformer-based Conditional Variational Autoencoder for Controllable Story Generation](https://mlsanity.com/paper/2101.00828/) (2021)
- [Variational Transformers for Diverse Response Generation](https://mlsanity.com/paper/2003.12738/) (2020)
- [A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text](https://mlsanity.com/paper/1909.00868/) (2019)
- [Lifelong Language Knowledge Distillation](https://mlsanity.com/paper/2010.02123/) (2020)
- [Continual Learning for Sentence Representations Using Conceptors](https://mlsanity.com/paper/1904.09187/) (2019)
- [Lifelong Learning for Sentiment Classification](https://mlsanity.com/paper/1801.02808/) (2018)
- [A Batch Normalized Inference Network Keeps the KL Vanishing Away](https://mlsanity.com/paper/2004.12585/) (2020)
- [Non-Autoregressive Machine Translation with Disentangled Context Transformer](https://mlsanity.com/paper/2001.05136/) (2020)
- [Variational Semi-supervised Aspect-term Sentiment Analysis via Transformer](https://mlsanity.com/paper/1810.10437/) (2018)
- [Reminding the Incremental Language Model via Data-Free Self-Distillation](https://mlsanity.com/paper/2110.08745/) (2021)

## Built on by (most cited first, 4 of 4)

- [Recent Advances of Foundation Language Models-based Continual Learning: A Survey](https://mlsanity.com/paper/2405.18653/) (2024)
- [Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback](https://mlsanity.com/paper/2505.09925/) (2025)
- [LSEBMCL: A Latent Space Energy-Based Model for Continual Learning](https://mlsanity.com/paper/2501.05495/) (2025)
- [PPSEBM: An Energy-Based Model with Progressive Parameter Selection for Continual Learning](https://mlsanity.com/paper/2512.15658/) (2025)

## Abstract (arXiv)

Lifelong Language Learning (LLL) aims to train a neural network to learn a stream of NLP tasks while retaining knowledge from previous tasks. However, previous works which followed data-free constraint still suffer from catastrophic forgetting issue, where the model forgets what it just learned from previous tasks. In order to alleviate catastrophic forgetting, we propose the residual variational autoencoder (RVAE) to enhance LAMOL, a recent LLL model, by mapping different tasks into a limited unified semantic space. In this space, previous tasks are easy to be correct to their own distribution by pseudo samples. Furthermore, we propose an identity task to make the model is discriminative to recognize the sample belonging to which task. For training RVAE-LAMOL better, we propose a novel training scheme Alternate Lag Training. In the experiments, we test RVAE-LAMOL on permutations of three datasets from DecaNLP. The experimental results demonstrate that RVAE-LAMOL outperforms na"ive LAMOL on all permutations and generates more meaningful pseudo-samples.