mlsanityData to 7 Oct 2026

PoF: Post-Training of Feature Extractor for Improving Generalization

arXiv 2207.01847 ↗First posted 5 Jul 2022cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

It has been intensively investigated that the local shape, especially flatness, of the loss landscape near a minimum plays an important role for generalization of deep models. We developed a training algorithm called PoF: Post-Training of Feature Extractor that updates the feature extractor part of an already-trained deep model to search a flatter minimum. The characteristics are two-fold: 1) Feature extractor is trained under parameter perturbations in the higher-layer parameter space, based on observations that suggest flattening higher-layer parameter space, and 2) the perturbation range is determined in a data-driven manner aiming to reduce a part of test loss caused by the positive loss curvature. We provide a theoretical analysis that shows the proposed algorithm implicitly reduces the target Hessian components as well as the loss. Experimental results show that PoF improved model performance against baseline methods on both CIFAR-10 and CIFAR-100 datasets for only 10-epoch post-training, and on SVHN dataset for 50-epoch post-training. Source code is available at: urlhttps://github.com/DensoITLab/PoF-v1

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 29 of its 47 references that are in the map. First posted on arXiv 5 Jul 2022.

Builds on

29 of 47 references.

References · 29

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  2. 2016Wide Residual Networkscited by 3,057
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  7. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  9. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  11. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  12. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
17 more
  1. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  2. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  3. 2014Qualitatively characterizing neural network optimization problemscited by 186
  4. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  5. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  6. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  7. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  8. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  9. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
  10. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  11. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  12. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  13. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 50
  14. 2020ESAM: Discriminative Domain Adaptation with Non-Displayed Items to Improve Long-Tail Performancecited by 30
  15. 2018Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learningcited by 20
  16. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  17. 2019Breaking Inter-Layer Co-Adaptation by Classifier Anonymizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Adaptive Adversarial Cross-Entropy Loss for Sharpness-Aware Minimizationcited by 1
  2. 2026EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbationscited by 0