# Cyclical Focal Loss

arXiv: https://arxiv.org/abs/2202.08978 · HTML: https://mlsanity.com/paper/2202.08978/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 21 of 21)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [TResNet: High Performance GPU-Dedicated Architecture](https://mlsanity.com/paper/2003.13630/) (2020)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [Three Mechanisms of Weight Decay Regularization](https://mlsanity.com/paper/1810.12281/) (2018)
- [Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence](https://mlsanity.com/paper/1905.13277/) (2019)
- [On the training dynamics of deep networks with L2 regularization](https://mlsanity.com/paper/2006.08643/) (2020)
- [Adaptive Weight Decay for Deep Neural Networks](https://mlsanity.com/paper/1907.08931/) (2019)
- [Understanding Decoupled and Early Weight Decay](https://mlsanity.com/paper/2012.13841/) (2020)
- [Focal Loss in 3D Object Detection](https://mlsanity.com/paper/1809.06065/) (2018)
- [Contrastive Unpaired Translation using Focal Loss for Patch Classification](https://mlsanity.com/paper/2109.12431/) (2021)

## Built on by (most cited first, 8 of 8)

- [Are All Losses Created Equal: A Neural Collapse Perspective](https://mlsanity.com/paper/2210.02192/) (2022)
- [A Short Survey on Importance Weighting for Machine Learning](https://mlsanity.com/paper/2403.10175/) (2024)
- [Neural Collapse in Multi-label Learning with Pick-all-label Loss](https://mlsanity.com/paper/2310.15903/) (2023)
- [Skeleton-based Action Recognition through Contrasting Two-Stream Spatial-Temporal Networks](https://mlsanity.com/paper/2301.11495/) (2023)
- [General Cyclical Training of Neural Networks](https://mlsanity.com/paper/2202.08835/) (2022)
- [Building Age Estimation: A New Multi-Modal Benchmark Dataset and Community Challenge](https://mlsanity.com/paper/2502.13818/) (2025)
- [A dual-branch model with inter- and intra-branch contrastive loss for long-tailed recognition](https://mlsanity.com/paper/2309.16135/) (2023)
- [MIFI: MultI-camera Feature Integration for Roust 3D Distracted Driver Activity Recognition](https://mlsanity.com/paper/2401.14115/) (2024)

## Abstract (arXiv)

The cross-entropy softmax loss is the primary loss function used to train deep neural networks. On the other hand, the focal loss function has been demonstrated to provide improved performance when there is an imbalance in the number of training samples in each class, such as in long-tailed datasets. In this paper, we introduce a novel cyclical focal loss and demonstrate that it is a more universal loss function than cross-entropy softmax loss or focal loss. We describe the intuition behind the cyclical focal loss and our experiments provide evidence that cyclical focal loss provides superior performance for balanced, imbalanced, or long-tailed datasets. We provide numerous experimental results for CIFAR-10/CIFAR-100, ImageNet, balanced and imbalanced 4,000 training sample versions of CIFAR-10/CIFAR-100, and ImageNet-LT and Places-LT from the Open Long-Tailed Recognition (OLTR) challenge. Implementing the cyclical focal loss function requires only a few lines of code and does not increase training time. In the spirit of reproducibility, our code is available at urlhttps://github.com/lnsmith54/CFL.