mlsanityData to 7 Oct 2026

Loss Functions and Operators Generated by f-Divergences

arXiv 2501.18537 ↗First posted 30 Jan 2025cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

The logistic loss (a.k.a. cross-entropy loss) is one of the most popular loss functions used for multiclass classification. It is also the loss function of choice for next-token prediction in language modeling. It is associated with the Kullback--Leibler (KL) divergence and the softargmax operator. In this work, we propose to construct new convex loss functions based on f-divergences. Our loss functions generalize the logistic loss in two directions: i) by replacing the KL divergence with f-divergences and ii) by allowing non-uniform reference measures. We instantiate our framework for numerous f-divergences, recovering existing losses and creating new ones. By analogy with the logistic loss, the loss function generated by an f-divergence is associated with an operator, that we dub f-softargmax. We derive a novel parallelizable bisection algorithm for computing the f-softargmax associated with any f-divergence. On the empirical side, one of the goals of this paper is to determine the effectiveness of loss functions beyond the classical cross-entropy in a language model setting, including on pre-training, post-training (SFT) and distillation. We show that the loss function generated by the α-divergence (which is equivalent to Tsallis α-negentropy in the case of unit reference measures) with α=1.5 performs well across several tasks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

25 of 44 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  11. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  12. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
13 more
  1. 2021Efficient and Modular Implicit Differentiationcited by 199
  2. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  3. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  4. 2019Sparse Sequence-to-Sequence Modelscited by 167
  5. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  6. 2019Learning with Fenchel-Young Lossescited by 117
  7. 2016R'enyi Divergence Variational Inferencecited by 104
  8. 2024The Elements of Differentiable Programmingcited by 41
  9. 2017f-Divergence constrained policy improvementcited by 15
  10. 2021Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivitycited by 11
  11. 2024f-Divergence Based Classification: Beyond the Use of Cross-Entropycited by 8
  12. 2021Moreau-Yosida f-divergencescited by 5
  13. 2024Robust Semi-supervised Learning via f-Divergence and α-R'enyi Divergencecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Predictioncited by 7
  2. 2025Establishing Linear Surrogate Regret Bounds for Convex Smooth Losses via Convolutional Fenchel-Young Lossescited by 2
  3. 2025A Unified Framework for Diffusion Model Unlearning with f-Divergencecited by 2
  4. 2025Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Lossescited by 1
  5. 2026Density-Ratio Losses for Post-Hoc Learning to Defercited by 1
  6. 2025Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Directioncited by 0
  7. 2025Alpha Divergence Losses for Biometric Verificationcited by 0
  8. 2026REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiencycited by 0
  9. 2026f-FUM: Federated Unlearning via min--max and f-divergencecited by 0
  10. 2026Sparsity-Inducing Divergence Losses for Biometric Verificationcited by 0