# A systematic study of the foreground-background imbalance problem in deep learning for object detection

arXiv: https://arxiv.org/abs/2306.16539 · HTML: https://mlsanity.com/paper/2306.16539/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 28 Jun 2023.

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [Detection and Tracking Meet Drones Challenge](https://mlsanity.com/paper/2001.06303/) (2020)
- [Libra R-CNN: Towards Balanced Learning for Object Detection](https://mlsanity.com/paper/1904.02701/) (2019)
- [Imbalance Problems in Object Detection: A Review](https://mlsanity.com/paper/1909.00169/) (2019)
- [Gradient Harmonized Single-stage Detector](https://mlsanity.com/paper/1811.05181/) (2018)
- [Perceptual Generative Adversarial Networks for Small Object Detection](https://mlsanity.com/paper/1706.05274/) (2017)
- [Prime Sample Attention in Object Detection](https://mlsanity.com/paper/1904.04821/) (2019)
- [Small-Object Detection in Remote Sensing Images with End-to-End Edge-Enhanced GAN and Object Detector Network](https://mlsanity.com/paper/2003.09085/) (2020)
- [Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-based CT Image Augmentation for Object Detection](https://mlsanity.com/paper/1906.04962/) (2019)
- [DR Loss: Improving Object Detection by Distributional Ranking](https://mlsanity.com/paper/1907.10156/) (2019)
- [AP-Loss for Accurate One-Stage Object Detection](https://mlsanity.com/paper/2008.07294/) (2020)
- [Foreground-Background Imbalance Problem in Deep Object Detectors: A Review](https://mlsanity.com/paper/2006.09238/) (2020)
- [Residual Objectness for Imbalance Reduction](https://mlsanity.com/paper/1908.09075/) (2019)
- [Unsupervised anomaly localization in high-resolution breast scans using deep pluralistic image completion](https://mlsanity.com/paper/2305.03098/) (2023)

## Built on by (most cited first, 3 of 3)

- [Exponentially Weighted Instance-Aware Repeat Factor Sampling for Long-Tailed Object Detection Model Training in Unmanned Aerial Vehicles Surveillance Scenarios](https://mlsanity.com/paper/2503.21893/) (2025)
- [An Empirical Study on the Robustness of YOLO Models for Underwater Object Detection](https://mlsanity.com/paper/2509.17561/) (2025)
- [Intelligent Detection of Mechanical, Electrical, and Plumbing (MEP) Metrics Based on 2D Floor Plans](https://mlsanity.com/paper/2608.14317/) (2026)

## Abstract (arXiv)

The class imbalance problem in deep learning has been explored in several studies, but there has yet to be a systematic analysis of this phenomenon in object detection. Here, we present comprehensive analyses and experiments of the foreground-background (F-B) imbalance problem in object detection, which is very common and caused by small, infrequent objects of interest. We experimentally study the effects of different aspects of F-B imbalance (object size, number of objects, dataset size, object type) on detection performance. In addition, we also compare 9 leading methods for addressing this problem, including Faster-RCNN, SSD, OHEM, Libra-RCNN, Focal-Loss, GHM, PISA, YOLO-v3, and GFL with a range of datasets from different imaging domains. We conclude that (1) the F-B imbalance can indeed cause a significant drop in detection performance, (2) The detection performance is more affected by F-B imbalance when fewer training data are available, (3) in most cases, decreasing object size leads to larger performance drop than decreasing number of objects, given the same change in the ratio of object pixels to non-object pixels, (6) among all selected methods, Libra-RCNN and PISA demonstrate the best performance in addressing the issue of F-B imbalance. (7) When the training dataset size is large, the choice of method is not impactful (8) Soft-sampling methods, including focal-loss, GHM, and GFL, perform fairly well on average but are relatively unstable.