Data to 7 Oct 2026

Improving Regression Performance with Distributional Losses

arXiv 1806.04613 ↗First posted 12 Jun 2018stat.ML cs.LGcited by 66

Not on the map. Why?

Abstract (arXiv)

There is growing evidence that converting targets to soft targets in supervised learning can provide considerable gains in performance. Much of this work has considered classification, converting hard zero-one values to soft labels---such as by adding label noise, incorporating label ambiguity or using distillation. In parallel, there is some evidence from a regression setting in reinforcement learning that learning distributions can improve performance. In this work, we investigate the reasons for this improvement, in a regression setting. We introduce a novel distributional regression loss, and similarly find it significantly improves prediction accuracy. We investigate several common hypotheses, around reducing overfitting and improved representations. We instead find evidence for an alternative hypothesis: this loss is easier to optimize, with better behaved gradients, resulting in improved generalization. We provide theoretical support for this alternative hypothesis, by characterizing the norm of the gradients of this loss.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 4; 2023: 5; 2024: 16; 2025: 22; 2026: 13.

66 papers.

Sort:

By area:

2023Mastering Diverse Domains through World Modelscited by 1,193
2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
2021Human Pose Regression with Residual Log-likelihood Estimationcited by 131
2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
2025Horizon Reduction Makes RL Scalablecited by 55
2024Amortized Planning with Large-Scale Transformers: A Case Study on Chesscited by 45
2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 43
2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
2024Efficient World Models with Context-Aware Tokenizationcited by 33
2023Distributional GFlowNets with Quantile Flowscited by 28
2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24

38 more

2024FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher informationcited by 17
2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 13
2023Combining Behaviors with the Successor Features Keyboardcited by 12
2023Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysiscited by 12
2021Exploring the Training Robustness of Distributional Reinforcement Learning against Noisy State Observationscited by 11
2024Investigating the Histogram Loss in Regressioncited by 11
2025Decoding-based Regressioncited by 11
2024Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Modelscited by 10
2024The Central Role of the Loss Function in Reinforcement Learningcited by 8
2024Winner-takes-all learners are geometry-aware conditional density estimatorscited by 6
2025Relative Entropy Pathwise Policy Optimizationcited by 6
2021Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learningcited by 4
2022How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?cited by 4
2024Predictive representations: building blocks of intelligencecited by 4
2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
2025Trajectory World Models for Heterogeneous Environmentscited by 4
2026Unified Multimodal Uncertain Inferencecited by 3
2021Sample Efficient Learning of Image-Based Diagnostic Classifiers Using Probabilistic Labelscited by 2
2021Ensemble and Auxiliary Tasks for Data-Efficient Deep Reinforcement Learningcited by 2
2022Distributional loss for convolutional neural network regression and application to GNSS multi-path estimationcited by 2
2022Neural Regression For Scale-Varying Targetscited by 2
2025The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibrationcited by 2
2025NetBurst: Event-Centric Forecasting of Bursty, Intermittent Time Seriescited by 2
2025Planning in Branch-and-Bound: Model-Based Reinforcement Learning for Exact Combinatorial Optimizationcited by 2
2026SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flowscited by 2
2026Start Classifying: Categorical Critics for LLM Reinforcement Learningcited by 2
2022Leveraging unsupervised data and domain adaptation for deep regression in low-cost sensor calibrationcited by 1
2025Building Bridges between Regression, Clustering, and Classificationcited by 1
2025Regularization of ML models for Earth systems by using longer model timestepscited by 1
2025Efficient and Scalable Estimation of Distributional Treatment Effects with Multi-Task Neural Networkscited by 1
2025Rectifying Regression in Reinforcement Learningcited by 1
2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
2025Understanding and Improving Hyperbolic Deep Reinforcement Learningcited by 1
2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
2021Disturbing Target Values for Neural Network Regularizationcited by 0
2024Distribution Learning for Molecular Regressioncited by 0

50 most cited of 66 All 66 (JSON)