mlsanityData to 7 Oct 2026

Zeroth-Order Sharpness-Aware Learning with Exponential Tilting

arXiv 2510.16157 ↗First posted 17 Oct 2025cs.LG cs.CL stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Classic zeroth-order optimization approaches typically optimize for a smoothed version of the original function, i.e., the expected objective under randomly perturbed model parameters. This can be interpreted as encouraging the loss values in the perturbation set to be small on average. Popular sharpness-aware minimization (SAM) objectives, however, typically focus on the largest loss within the neighborhood to arrive at flat minima more effectively. In this work, we connect zeroth-order optimization (and its corresponding objectives) with SAM approaches explicitly, through an exponential tilting objective that provides a smooth transition between the average- and the max-loss formulations. We explore new zeroth-order algorithms to solve a soft SAM objective parameterized by a tilting parameter t. We provide precise characterizations of the sharpness notions of the tilted SAM framework. Practically, our approach can be used as a gradient-free and memory-efficient alternative to SAM variants, and it achieves better generalization compared to vanilla zeroth-order baselines on a wide range of downstream tasks, including classification, multiple choice QA, and language generation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

32 of 45 references.

References · 32

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  6. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  7. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  8. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
  9. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  10. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  11. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  12. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
20 more
  1. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  2. 2015An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedbackcited by 105
  3. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  4. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  5. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  6. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  7. 2021On Tilted Losses in Machine Learning: Theory and Applicationscited by 42
  8. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  9. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  10. 2022Probabilistically Robust Learning: Balancing Average- and Worst-case Performancecited by 36
  11. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  12. 2016Highly-Smooth Zero-th Order Online Optimization Vianney Perchetcited by 22
  13. 2024Why is SAM Robust to Label Noise?cited by 21
  14. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 16
  15. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  16. 2023TRAM: Bridging Trust Regions and Sharpness Aware Minimizationcited by 6
  17. 2024Tilted Sharpness-Aware Minimizationcited by 6
  18. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  19. 2024Generalization and Robustness of the Tilted Empirical Riskcited by 4
  20. 2024Sharpness-Aware Black-Box Optimizationcited by 2

Built on by

Not cited yet.