# Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection

arXiv: https://arxiv.org/abs/2307.12427 · HTML: https://mlsanity.com/paper/2307.12427/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 68 references that are in the map.
- First posted on arXiv 23 Jul 2023.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)

## Built on by (most cited first, 30 of 32)

- [FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning](https://mlsanity.com/paper/2309.14062/) (2023)
- [Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning](https://mlsanity.com/paper/2405.19074/) (2024)
- [VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model](https://mlsanity.com/paper/2403.05346/) (2024)
- [Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting](https://mlsanity.com/paper/2508.04227/) (2025)
- [Bridge Past and Future: Overcoming Information Asymmetry in Incremental Object Detection](https://mlsanity.com/paper/2407.11499/) (2024)
- [Is Parameter Collision Hindering Continual Learning in LLMs?](https://mlsanity.com/paper/2410.10179/) (2024)
- [Object Detectors in the Open Environment: Challenges, Solutions, and Outlook](https://mlsanity.com/paper/2403.16271/) (2024)
- [Calibrating Higher-Order Statistics for Few-Shot Class-Incremental Learning with Pre-trained Vision Transformers](https://mlsanity.com/paper/2404.06622/) (2024)
- [Preventing Catastrophic Forgetting through Memory Networks in Continuous Detection](https://mlsanity.com/paper/2403.14797/) (2024)
- [Multi-Label Continual Learning for the Medical Domain: A Novel Benchmark](https://mlsanity.com/paper/2404.06859/) (2024)
- [Replay Consolidation with Label Propagation for Continual Object Detection](https://mlsanity.com/paper/2409.05650/) (2024)
- [Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector](https://mlsanity.com/paper/2502.05540/) (2025)
- [RICO: Two Realistic Benchmarks and an In-Depth Analysis for Incremental Learning in Object Detection](https://mlsanity.com/paper/2508.13878/) (2025)
- [Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration](https://mlsanity.com/paper/2403.11373/) (2024)
- [CGL: Advancing Continual GUI Learning via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2603.02951/) (2026)
- [IOR: Inversed Objects Replay for Incremental Object Detection](https://mlsanity.com/paper/2406.04829/) (2024)
- [Incremental Object Detection with Prompt-based Methods](https://mlsanity.com/paper/2508.14599/) (2025)
- [Teach YOLO to Remember: A Self-Distillation Approach for Continual Object Detection](https://mlsanity.com/paper/2503.04688/) (2025)
- [DuET: Dual Incremental Object Detection via Exemplar-Free Task Arithmetic](https://mlsanity.com/paper/2506.21260/) (2025)
- [OW-CLIP: Data-Efficient Visual Supervision for Open-World Object Detection via Human-AI Collaboration](https://mlsanity.com/paper/2507.19870/) (2025)
- [Learning on the Fly: Replay-Based Continual Object Perception for Indoor Drones](https://mlsanity.com/paper/2602.13440/) (2026)
- [EW-DETR: Evolving World Object Detection via Incremental Low-Rank DEtection TRansformer](https://mlsanity.com/paper/2602.20985/) (2026)
- [STAR-IOD: Scale-decoupled Topology Alignment with Pseudo-label Refinement for Remote Sensing Incremental Object Detection](https://mlsanity.com/paper/2605.20738/) (2026)
- [Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection](https://mlsanity.com/paper/2606.15253/) (2026)
- [Symbiosis-Inspired Knowledge Distillation for Incremental Object Detection](https://mlsanity.com/paper/2607.13452/) (2026)
- [CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detection](https://mlsanity.com/paper/2410.05804/) (2024)
- [Exploiting Stability-Plasticity Asymmetry in Pretrained Detectors for Incremental Object Detection](https://mlsanity.com/paper/2504.10214/) (2025)
- [Incremental Human-Object Interaction Detection with Invariant Relation Representation Learning](https://mlsanity.com/paper/2510.27020/) (2025)
- [Parameterized Prompt for Incremental Object Detection](https://mlsanity.com/paper/2510.27316/) (2025)
- [Better Matching, Less Forgetting: A Quality-Guided Matcher for Transformer-based Incremental Object Detection](https://mlsanity.com/paper/2603.01524/) (2026)

## Abstract (arXiv)

In incremental learning, replaying stored samples from previous tasks together with current task samples is one of the most efficient approaches to address catastrophic forgetting. However, unlike incremental classification, image replay has not been successfully applied to incremental object detection (IOD). In this paper, we identify the overlooked problem of foreground shift as the main reason for this. Foreground shift only occurs when replaying images of previous tasks and refers to the fact that their background might contain foreground objects of the current task. To overcome this problem, a novel and efficient Augmented Box Replay (ABR) method is developed that only stores and replays foreground objects and thereby circumvents the foreground shift problem. In addition, we propose an innovative Attentive RoI Distillation loss that uses spatial attention from region-of-interest (RoI) features to constrain current model to focus on the most important information from old model. ABR significantly reduces forgetting of previous classes while maintaining high plasticity in current classes. Moreover, it considerably reduces the storage requirements when compared to standard image replay. Comprehensive experiments on Pascal-VOC and COCO datasets support the state-of-the-art performance of our model.