# Cheap Thrills: Effective Amortized Optimization Using Inexpensive Labels

arXiv: https://arxiv.org/abs/2603.05495 · HTML: https://mlsanity.com/paper/2603.05495/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks](https://mlsanity.com/paper/2207.10289/) (2022)
- [Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers](https://mlsanity.com/paper/2007.00016/) (2020)
- [DC3: A learning method for optimization with hard constraints](https://mlsanity.com/paper/2104.12225/) (2021)
- [Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios](https://mlsanity.com/paper/2212.11419/) (2022)
- [On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization](https://mlsanity.com/paper/1607.05447/) (2016)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [Lagrangian Duality for Constrained Deep Learning](https://mlsanity.com/paper/2001.09394/) (2020)
- [Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling](https://mlsanity.com/paper/2207.02338/) (2022)
- [Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities](https://mlsanity.com/paper/2409.03444/) (2024)
- [DeepOPF: A Feasibility-Optimized Deep Neural Network Approach for AC Optimal Power Flow Problems](https://mlsanity.com/paper/2007.01002/) (2020)
- [Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study](https://mlsanity.com/paper/1708.07827/) (2017)
- [Learning to Warm-Start Fixed-Point Optimization Algorithms](https://mlsanity.com/paper/2309.07835/) (2023)
- [RAYEN: Imposition of Hard Convex Constraints on Neural Networks](https://mlsanity.com/paper/2307.08336/) (2023)
- [CANOS: A Fast and Scalable Neural AC-OPF Solver Robust To N-1 Perturbations](https://mlsanity.com/paper/2403.17660/) (2024)
- [OPFData: Large-scale datasets for AC optimal power flow with topological perturbations](https://mlsanity.com/paper/2406.07234/) (2024)
- [FSNet: Feasibility-Seeking Neural Network for Constrained Optimization with Guarantees](https://mlsanity.com/paper/2506.00362/) (2025)
- [Pinet: Optimizing hard-constrained neural networks with orthogonal projection layers](https://mlsanity.com/paper/2508.10480/) (2025)
- [Optimization Learning](https://mlsanity.com/paper/2501.03443/) (2025)
- [Geometry and convergence of natural policy gradient methods](https://mlsanity.com/paper/2211.02105/) (2022)
- [PGLearn -- An Open-Source Learning Toolkit for Optimal Power Flow](https://mlsanity.com/paper/2505.22825/) (2025)
- [What Data Enables Optimal Decisions? An Exact Characterization for Linear Optimization](https://mlsanity.com/paper/2505.21692/) (2025)

## Built on by (most cited first, 1 of 1)

- [Scalable Self-Supervised Learning for Multiphase AC-OPF in Distribution Systems with Topology Reconfiguration](https://mlsanity.com/paper/2608.25095/) (2026)

## Abstract (arXiv)

To scale optimization and simulation, prior work has explored training machine-learning surrogates that map problem parameters to solutions inexpensively at inference time. Unfortunately, commonly used approaches, including supervised and self-supervised learning with either soft or hard feasibility enforcement, face inherent challenges such as reliance on expensive high-quality labels or difficult optimization landscapes. To address their trade-offs, we propose a novel framework that collects "cheap" imperfect labels, performs supervised model pretraining with a merit loss-based termination scheme, and finally refines the model through self-supervised learning to improve final performance. Empirical validation across challenging domains -- including nonconvex constrained optimization, power-grid operation, and stiff dynamical systems -- shows that this three-stage strategy yields faster convergence; improved accuracy, feasibility, and optimality; and up to 59x reductions in total offline computational cost. We further analyze why and when our framework improves surrogate model training, finding that (i) merit loss is an informative signal and (ii) only small numbers of cheap, inexact labels are needed to place the model in a favorable regime for self-supervised learning.