mlsanityData to 14 Oct 2026

Optimization with Access to Auxiliary Information

arXiv 2206.00395 ↗First posted 1 Jun 2022cs.LG math.OCcited by 13

Not on the map. Why?

Abstract (arXiv)

We investigate the fundamental optimization question of minimizing a target function f, whose gradients are expensive to compute or have limited availability, given access to some auxiliary side function h whose gradients are cheap or more available. This formulation captures many settings of practical relevance, such as i) re-using batches in SGD, ii) transfer learning, iii) federated learning, iv) training with compressed models/dropout, Et cetera. We propose two generic new algorithms that apply in all these settings; we also prove that we can benefit from this framework under the Hessian similarity assumption between the target and side information. A benefit is obtained when this similarity measure is small; we also show a potential benefit from stochasticity when the auxiliary noise is correlated with that of the target function.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 1 Jun 2022.

Builds on

30 of 48 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  7. 2014How transferable are features in deep neural networks?cited by 1,225
  8. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  9. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  10. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  11. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  12. 2019Agnostic Federated Learningcited by 456
18 more
  1. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  2. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  3. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  4. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 220
  5. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 175
  6. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  7. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 145
  8. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 132
  9. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 119
  10. 2020On the Convergence of SGD with Biased Gradientscited by 83
  11. 2017Practical Coreset Constructions for Machine Learningcited by 78
  12. 2017Non-convex Finite-Sum Optimization Via SCSG Methodscited by 61
  13. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 59
  14. 2020Auxiliary Learning by Implicit Differentiationcited by 45
  15. 2017meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfittingcited by 34
  16. 2020Auxiliary Task Reweighting for Minimum-data Learningcited by 23
  17. 2021Masked Training of Neural Networks with Partial Gradientscited by 16
  18. 2021Linear Speedup in Personalized Collaborative Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 2; 2025: 4; 2026: 1.

13 papers.

Sort:

By area:

  1. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  2. 2022Faster federated optimization under second-order similaritycited by 18
  3. 2023Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methodscited by 12
  4. 2023Two Losses Are Better Than One: Faster Optimization Using a Cheaper Proxycited by 7
  5. 2024Stabilized Proximal-Point Methods for Federated Optimizationcited by 7
  6. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  7. 2023Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correctioncited by 5
  8. 2024SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learningcited by 3
  9. 2025Stochastic Difference-of-Convex Optimization with Momentumcited by 2
  10. 2025Stochastic Optimization with Random Searchcited by 1
  11. 2025Non-Convex Federated Optimization under Cost-Aware Client Selectioncited by 1
  12. 2025Second-Order Optimization with Asynchronous Curvaturecited by 0
1 more
  1. 2026Partially Informed Stochastic Optimization via Zeroth-Order Residual Trackingcited by 0