# f-Divergence Based Classification: Beyond the Use of Cross-Entropy

arXiv: https://arxiv.org/abs/2401.01268 · HTML: https://mlsanity.com/paper/2401.01268/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 2 Jan 2024.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks](https://mlsanity.com/paper/2006.07322/) (2020)
- [Learning in Implicit Generative Models](https://mlsanity.com/paper/1610.03483/) (2016)
- [Deep Learning-Based Channel Estimation](https://mlsanity.com/paper/1810.05893/) (2018)
- [Deep Learning Methods for Improved Decoding of Linear Codes](https://mlsanity.com/paper/1706.07043/) (2017)
- [When Optimizing f-divergence is Robust with Label Noise](https://mlsanity.com/paper/2011.03687/) (2020)
- [Training Deep Energy-Based Models with f-Divergence Minimization](https://mlsanity.com/paper/2003.03463/) (2020)
- [Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning](https://mlsanity.com/paper/1812.06589/) (2018)
- [Mutual Information Estimation via f-Divergence and Data Derangements](https://mlsanity.com/paper/2305.20025/) (2023)
- [Copula Density Neural Estimation](https://mlsanity.com/paper/2211.15353/) (2022)
- [Learning Fair Classifiers via Min-Max F-divergence Regularization](https://mlsanity.com/paper/2306.16552/) (2023)

## Built on by (most cited first, 8 of 8)

- [Loss Functions and Operators Generated by f-Divergences](https://mlsanity.com/paper/2501.18537/) (2025)
- [Robust Classification with Noisy Labels Based on Posterior Maximization](https://mlsanity.com/paper/2504.06805/) (2025)
- [A Unified Framework for Diffusion Model Unlearning with f-Divergence](https://mlsanity.com/paper/2509.21167/) (2025)
- [Alpha Divergence Losses for Biometric Verification](https://mlsanity.com/paper/2511.13621/) (2025)
- [The Multiclass Score-Oriented Loss (MultiSOL) on the Simplex](https://mlsanity.com/paper/2511.22587/) (2025)
- [Empirical Risk Minimization with f-Divergence Regularization](https://mlsanity.com/paper/2601.13191/) (2026)
- [f-FUM: Federated Unlearning via min--max and f-divergence](https://mlsanity.com/paper/2602.06187/) (2026)
- [Sparsity-Inducing Divergence Losses for Biometric Verification](https://mlsanity.com/paper/2606.31664/) (2026)

## Abstract (arXiv)

In deep learning, classification tasks are formalized as optimization problems often solved via the minimization of the cross-entropy. However, recent advancements in the design of objective functions allow the usage of the f-divergence to generalize the formulation of the optimization problem for classification. We adopt a Bayesian perspective and formulate the classification task as a maximum a posteriori probability problem. We propose a class of objective functions based on the variational representation of the f-divergence. Furthermore, driven by the challenge of improving the state-of-the-art approach, we propose a bottom-up method that leads us to the formulation of an objective function corresponding to a novel f-divergence referred to as shifted log (SL). We theoretically analyze the objective functions proposed and numerically test them in three application scenarios: toy examples, image datasets, and signal detection/decoding problems. The analyzed scenarios demonstrate the effectiveness of the proposed approach and that the SL divergence achieves the highest classification accuracy in almost all the considered cases.