mlsanityData to 14 Oct 2026

Regularizing Neural Networks by Penalizing Confident Output Distributions

arXiv 1701.06548 ↗First posted 23 Jan 2017cs.NE cs.LGcited by 464

Not on the map. Why?

Cited by 21 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT'14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.

Key facts. Cited by 464 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Jan 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 102; 2022: 94; 2023: 87; 2024: 75; 2025: 58; 2026: 48.

464 papers.

Sort:

By area:

  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  2. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  3. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
  4. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  5. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 139
  6. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 119
  7. 2022Locally Typical Samplingcited by 111
  8. 2022Membership Inference Attacks by Exploiting Loss Trajectorycited by 101
  9. 2021The Devil is in the Margin: Margin-based Label Smoothing for Network Calibrationcited by 91
  10. 2021Augmentation Strategies for Learning with Noisy Labelscited by 86
  11. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  12. 2021Dynamic Slimmable Networkcited by 83
38 more
  1. 2021Robust Audio-Visual Instance Discriminationcited by 80
  2. 2021RobustART: Benchmarking Robustness on Architecture Design and Training Techniquescited by 79
  3. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 75
  4. 2021To Smooth or Not? When Label Smoothing Meets Noisy Labelscited by 70
  5. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  6. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  7. 2022PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functionscited by 63
  8. 2022Part-based Pseudo Label Refinement for Unsupervised Person Re-identificationcited by 59
  9. 2023Uncertainty Estimation by Fisher Information-based Evidential Deep Learningcited by 58
  10. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  11. 2022Robust Cross-Modal Representation Learning with Progressive Self-Distillationcited by 55
  12. 2021Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Studycited by 51
  13. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  14. 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
  15. 2021SmoothMix: Training Confidence-calibrated Smoothed Classifiers for Certified Robustnesscited by 49
  16. 2022A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibrationcited by 49
  17. 2021Sample Selection for Fair and Robust Trainingcited by 48
  18. 2022A Consistent and Differentiable Lp Canonical Calibration Error Estimatorcited by 47
  19. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  20. 2023Improving Deep Regression with Ordinal Entropycited by 45
  21. 2021BERT & Family Eat Word Salad: Experiments with Text Understandingcited by 44
  22. 2021Active label cleaning for improved dataset quality under resource constraintscited by 44
  23. 2023LitCab: Lightweight Language Model Calibration over Short- and Long-form Responsescited by 44
  24. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 44
  25. 2021One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objectivecited by 43
  26. 2022Eliciting and Learning with Soft Labels from Every Annotatorcited by 43
  27. 2023Rethinking Confidence Calibration for Failure Predictioncited by 43
  28. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  29. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  30. 2022RelaxLoss: Defending Membership Inference Attacks without Losing Utilitycited by 37
  31. 2024Knowledge Distillation Based on Transformed Teacher Matchingcited by 37
  32. 2021Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safetycited by 35
  33. 2021Meta-Cal: Well-controlled Post-hoc Calibration by Rankingcited by 33
  34. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33
  35. 2022Self-Filtering: A Noise-Aware Sample Selection for Label Noise with Confidence Penalizationcited by 33
  36. 2022Self-Supervised Learning via Maximum Entropy Codingcited by 33
  37. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  38. 2022Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?cited by 32

50 most cited of 464 All 464 (JSON)