# Adaptive Model Ensemble for Continual Learning

arXiv: https://arxiv.org/abs/2509.19819 · HTML: https://mlsanity.com/paper/2509.19819/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [New Insights on Reducing Abrupt Representation Change in Online Continual Learning](https://mlsanity.com/paper/2104.05025/) (2021)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System](https://mlsanity.com/paper/2201.12604/) (2022)
- [Measuring and regularizing networks in function space](https://mlsanity.com/paper/1805.08289/) (2018)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [Gradient-based Editing of Memory Examples for Online Task-free Continual Learning](https://mlsanity.com/paper/2006.15294/) (2020)
- [Consistent Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2403.08568/) (2024)
- [PCR: Proxy-based Contrastive Replay for Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2304.04408/) (2023)
- [Exploring Example Influence in Continual Learning](https://mlsanity.com/paper/2209.12241/) (2022)
- [Learning to Forget for Meta-Learning](https://mlsanity.com/paper/1906.05895/) (2019)
- [Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning](https://mlsanity.com/paper/2405.19074/) (2024)
- [Convolutional Prompting meets Language Models for Continual Learning](https://mlsanity.com/paper/2403.20317/) (2024)
- [Meta-attention for ViT-backed Continual Learning](https://mlsanity.com/paper/2203.11684/) (2022)
- [On Generalizing Beyond Domains in Cross-Domain Continual Learning](https://mlsanity.com/paper/2203.03970/) (2022)
- [Regularizing Second-Order Influences for Continual Learning](https://mlsanity.com/paper/2304.10177/) (2023)
- [CBA: Improving Online Continual Learning via Continual Bias Adaptor](https://mlsanity.com/paper/2308.06925/) (2023)
- [Exploring Data Geometry for Continual Learning](https://mlsanity.com/paper/2304.03931/) (2023)
- [Improving Online Continual Learning Performance and Stability with Temporal Ensembles](https://mlsanity.com/paper/2306.16817/) (2023)
- [Latent Spectral Regularization for Continual Learning](https://mlsanity.com/paper/2301.03345/) (2023)
- [Preserving Linear Separability in Continual Learning by Backward Feature Projection](https://mlsanity.com/paper/2303.14595/) (2023)

## Abstract (arXiv)

Model ensemble is an effective strategy in continual learning, which alleviates catastrophic forgetting by interpolating model parameters, achieving knowledge fusion learned from different tasks. However, existing model ensemble methods usually encounter the knowledge conflict issue at task and layer levels, causing compromised learning performance in both old and new tasks. To solve this issue, we propose meta-weight-ensembler that adaptively fuses knowledge of different tasks for continual learning. Concretely, we employ a mixing coefficient generator trained via meta-learning to generate appropriate mixing coefficients for model ensemble to address the task-level knowledge conflict. The mixing coefficient is individually generated for each layer to address the layer-level knowledge conflict. In this way, we learn the prior knowledge about adaptively accumulating knowledge of different tasks in a fused model, achieving efficient learning in both old and new tasks. Meta-weight-ensembler can be flexibly combined with existing continual learning methods to boost their ability of alleviating catastrophic forgetting. Experiments on multiple continual learning datasets show that meta-weight-ensembler effectively alleviates catastrophic forgetting and achieves state-of-the-art performance.