mlsanityData to 7 Oct 2026

Investigating the Histogram Loss in Regression

arXiv 2402.13425 ↗First posted 20 Feb 2024cs.LG cs.AI stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

It is becoming increasingly common in regression to train neural networks that model the entire distribution even if only the mean is required for prediction. This additional modeling often comes with performance gain and the reasons behind the improvement are not fully known. This paper investigates a recent approach to regression, the Histogram Loss, which involves learning the conditional distribution of the target variable by minimizing the cross-entropy between a target distribution and a flexible histogram prediction. We design theoretical and empirical analyses to determine why and when this performance gain appears, and how different components of the loss contribute to it. Our results suggest that the benefits of learning distributions in this setup come from improvements in optimization rather than modelling extra information. We then demonstrate the viability of the Histogram Loss in common deep learning applications without a need for costly hyperparameter tuning.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 32 of its 65 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

32 of 65 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  10. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  11. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  12. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
20 more
  1. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  2. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  3. 2021Categorical Depth Distribution Network for Monocular 3D Object Detectioncited by 326
  4. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  5. 2014The Loss Surfaces of Multilayer Networkscited by 278
  6. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  7. 2017A General and Adaptive Robust Loss Functioncited by 236
  8. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  9. 2017PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparisoncited by 139
  10. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  11. 2016Robust Large Margin Deep Neural Networkscited by 117
  12. 2016Deep Label Distribution Learning with Label Ambiguitycited by 104
  13. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  14. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  15. 2018Improving Regression Performance with Distributional Lossescited by 66
  16. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  17. 2019ChainerRL: A Deep Reinforcement Learning Librarycited by 51
  18. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  19. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  20. 2023Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networkscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 2.

11 papers.

  1. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  2. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  3. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 12
  4. 2025Open-ended Hierarchical Streaming Video Understanding with Vision Language Modelscited by 6
  5. 2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
  6. 2026Unified Multimodal Uncertain Inferencecited by 3
  7. 2025Modeling Gene Expression Distributional Shifts for Unseen Genetic Perturbationscited by 1
  8. 2025Rectifying Regression in Reinforcement Learningcited by 1
  9. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
  10. 2025D2 Actor Critic: Diffusion Actor Meets Distributional Criticcited by 0
  11. 2026Learning the Supports for Categorical Critic in Reinforcement Learningcited by 0