# Forgetting to Remember: A Scalable Incremental Learning Framework for Cross-Task Blind Image Quality Assessment

arXiv: https://arxiv.org/abs/2209.07126 · HTML: https://mlsanity.com/paper/2209.07126/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 80 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment](https://mlsanity.com/paper/1910.06180/) (2019)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Network](https://mlsanity.com/paper/1907.02665/) (2019)
- [Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment](https://mlsanity.com/paper/1612.01697/) (2016)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)
- [Massive Online Crowdsourced Study of Subjective and Objective Picture Quality](https://mlsanity.com/paper/1511.02919/) (2015)
- [Compacting, Picking and Growing for Unforgetting Continual Learning](https://mlsanity.com/paper/1910.06562/) (2019)
- [MetaIQA: Deep Meta-learning for No-Reference Image Quality Assessment](https://mlsanity.com/paper/2004.05508/) (2020)
- [Uncertainty-guided Continual Learning with Bayesian Neural Networks](https://mlsanity.com/paper/1906.02425/) (2019)
- [Continual Learning for Blind Image Quality Assessment](https://mlsanity.com/paper/2102.09717/) (2021)
- [LIQA: Lifelong Blind Image Quality Assessment](https://mlsanity.com/paper/2104.14115/) (2021)
- [Task-Specific Normalization for Continual Learning of Blind Image Quality Models](https://mlsanity.com/paper/2107.13429/) (2021)
- [Learning to Blindly Assess Image Quality in the Laboratory and Wild](https://mlsanity.com/paper/1907.00516/) (2019)
- [A Perceptually Weighted Rank Correlation Indicator for Objective Image Quality Assessment](https://mlsanity.com/paper/1705.05126/) (2017)

## Built on by (most cited first, 4 of 4)

- [Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning](https://mlsanity.com/paper/2301.10931/) (2023)
- [Video Quality Assessment: A Comprehensive Survey](https://mlsanity.com/paper/2412.04508/) (2024)
- [Life-IQA: Boosting Blind Image Quality Assessment through GCN-enhanced Layer Interaction and MoE-based Feature Decoupling](https://mlsanity.com/paper/2511.19024/) (2025)
- [Distortion-Resilient Robotic Imitation Learning for Autonomous Cable Routing](https://mlsanity.com/paper/2606.11577/) (2026)

## Abstract (arXiv)

Recent years have witnessed the great success of blind image quality assessment (BIQA) in various task-specific scenarios, which present invariable distortion types and evaluation criteria. However, due to the rigid structure and learning framework, they cannot apply to the cross-task BIQA scenario, where the distortion types and evaluation criteria keep changing in practical applications. This paper proposes a scalable incremental learning framework (SILF) that could sequentially conduct BIQA across multiple evaluation tasks with limited memory capacity. More specifically, we develop a dynamic parameter isolation strategy to sequentially update the task-specific parameter subsets, which are non-overlapped with each other. Each parameter subset is temporarily settled to Remember one evaluation preference toward its corresponding task, and the previously settled parameter subsets can be adaptively reused in the following BIQA to achieve better performance based on the task relevance. To suppress the unrestrained expansion of memory capacity in sequential tasks learning, we develop a scalable memory unit by gradually and selectively pruning unimportant neurons from previously settled parameter subsets, which enable us to Forget part of previous experiences and free the limited memory capacity for adapting to the emerging new tasks. Extensive experiments on eleven IQA datasets demonstrate that our proposed method significantly outperforms the other state-of-the-art methods in cross-task BIQA. The source code of the proposed method is available at https://github.com/maruiperfect/SILF.