mlsanityData to 7 Oct 2026

Two Losses Are Better Than One: Faster Optimization Using a Cheaper Proxy

arXiv 2302.03542 ↗First posted 7 Feb 2023cs.LG math.OCcited by 7

Not on the map. Why?

Abstract (arXiv)

We present an algorithm for minimizing an objective with hard-to-compute gradients by using a related, easier-to-access function as a proxy. Our algorithm is based on approximate proximal point iterations on the proxy combined with relatively few stochastic gradients from the objective. When the difference between the objective and the proxy is δ-smooth, our algorithm guarantees convergence at a rate matching stochastic gradient descent on a δ-smooth objective, which can lead to substantially better sample efficiency. Our algorithm has many potential applications in machine learning, and provides a principled means of leveraging synthetic data, physics simulators, mixed public and private data, and more.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

20 of 36 references.

References · 20

  1. 2016OpenAI Gymcited by 1,684
  2. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  3. 2017A Survey on Multi-Task Learningcited by 838
  4. 2014Convex Optimization: Algorithms and Complexitycited by 569
  5. 2018Do Better ImageNet Models Transfer Better?cited by 560
  6. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  7. 2018Rethinking ImageNet Pre-trainingcited by 376
  8. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  9. 2018Synthetic Data Augmentation using GAN for Improved Liver Lesion Classificationcited by 136
  10. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 131
  11. 2015Communication Complexity of Distributed Convex Learning and Optimizationcited by 59
  12. 2018How To Make the Gradients Small Stochastically: Even Faster Convex and Nonconvex SGDcited by 55
8 more
  1. 2014Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learningcited by 35
  2. 2019The Complexity of Making the Gradient Small in Stochastic Convex Optimizationcited by 33
  3. 2020Statistically Preconditioned Accelerated Gradient Method for Distributed Optimizationcited by 29
  4. 2013Stochastic Majorization-Minimization Algorithms for Large-Scale Optimizationcited by 28
  5. 2021Fast Stochastic Bregman Gradient Methods: Sharp Analysis and Variance Reductioncited by 19
  6. 2022Optimization with Access to Auxiliary Informationcited by 12
  7. 2022Optimal Gradient Sliding and its Application to Distributed Optimization Under Similaritycited by 9
  8. 2020Principled Analyses and Design of First-Order Methods with Inexact Proximal Operatorscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 1; 2026: 2.

7 papers.

  1. 2023Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methodscited by 10
  2. 2024Non-convex Stochastic Composite Optimization with Polyak Momentumcited by 9
  3. 2024SPAM: Stochastic Proximal Point Method with Momentum Variance Reduction for Non-convex Cross-Device Federated Learningcited by 3
  4. 2024Accelerated Methods with Compressed Communications for Distributed Optimization Problems under Data Similaritycited by 2
  5. 2025Simple Optimizers for Convex Aligned Multi-Objective Optimizationcited by 1
  6. 2026Accelerated Methods with Complexity Separation Under Data Similarity for Federated Learning Problemscited by 0
  7. 2026From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similaritycited by 0