# SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions

arXiv: https://arxiv.org/abs/1912.12355 · HTML: https://mlsanity.com/paper/1912.12355/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2019.

## Built on by (most cited first, 30 of 40)

- [Multi-Objective Loss Balancing for Physics-Informed Deep Learning](https://mlsanity.com/paper/2110.09813/) (2021)
- [Human-inspired Perspectives: A Survey on AI Long-term Memory](https://mlsanity.com/paper/2411.00489/) (2024)
- [DeepGD: A Deep Learning Framework for Graph Drawing Using GNN](https://mlsanity.com/paper/2106.15347/) (2021)
- [Preoperative brain tumor imaging: models and software for segmentation and standardized reporting](https://mlsanity.com/paper/2204.14199/) (2022)
- [QCPINN: Quantum-Classical Physics-Informed Neural Networks for Solving PDEs](https://mlsanity.com/paper/2503.16678/) (2025)
- [Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equations](https://mlsanity.com/paper/2411.15111/) (2024)
- [Enabling Automatic Differentiation with Mollified Graph Neural Operators](https://mlsanity.com/paper/2504.08277/) (2025)
- [Physics-informed neural networks modeling for systems with moving immersed boundaries: application to an unsteady flow past a plunging foil](https://mlsanity.com/paper/2306.13395/) (2023)
- [Domain-decoupled Physics-informed Neural Networks with Closed-form Gradients for Fast Model Learning of Dynamical Systems](https://mlsanity.com/paper/2408.14951/) (2024)
- [Physics-Guided Neural Networks for Intraventricular Vector Flow Mapping](https://mlsanity.com/paper/2403.13040/) (2024)
- [OWAdapt: An adaptive loss function for deep learning using OWA operators](https://mlsanity.com/paper/2305.19443/) (2023)
- [Understanding the Effect of the Long Tail on Neural Network Compression](https://mlsanity.com/paper/2306.06238/) (2023)
- [Adaptive Real-Time Multi-Loss Function Optimization Using Dynamic Memory Fusion Framework: A Case Study on Breast Cancer Segmentation](https://mlsanity.com/paper/2410.19745/) (2024)
- [Physics-Informed Regression: Parameter Estimation in Parameter-Linear Nonlinear Dynamic Models](https://mlsanity.com/paper/2508.19249/) (2025)
- [SD-RetinaNet: Topologically Constrained Semi-Supervised Retinal Lesion and Layer Segmentation in OCT](https://mlsanity.com/paper/2509.20864/) (2025)
- [Applied monocular reconstruction of parametric faces with domain engineering](https://mlsanity.com/paper/2208.02935/) (2022)
- [VI-PINNs: Variance-involved Physics-informed Neural Networks for Fast and Accurate Prediction of Partial Differential Equations](https://mlsanity.com/paper/2211.16753/) (2022)
- [On Training Derivative-Constrained Neural Networks](https://mlsanity.com/paper/2310.01649/) (2023)
- [DSAF: A Dual-Stage Adaptive Framework for Numerical Weather Prediction Downscaling](https://mlsanity.com/paper/2312.12476/) (2023)
- [Network scaling and scale-driven loss balancing for intelligent poroelastography](https://mlsanity.com/paper/2411.08886/) (2024)
- [Learning low-dimensional representations of ensemble forecast fields using autoencoder-based methods](https://mlsanity.com/paper/2502.04409/) (2025)
- [Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulation](https://mlsanity.com/paper/2507.16008/) (2025)
- [Bridging ocean wave physics and deep learning: Physics-informed neural operators for nonlinear wavefield reconstruction in real-time](https://mlsanity.com/paper/2508.03315/) (2025)
- [Physics-Informed Neural Operators for Cardiac Electrophysiology](https://mlsanity.com/paper/2511.08418/) (2025)
- [Enhancing Physics-Informed Neural Networks with Domain-aware Fourier Features: Towards Improved Performance and Interpretable Results](https://mlsanity.com/paper/2603.02948/) (2026)
- [Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks](https://mlsanity.com/paper/2605.10136/) (2026)
- [Domain Engineering for Applied Monocular Reconstruction of Parametric Faces](https://mlsanity.com/paper/2209.02600/) (2022)
- [An Operator Learning Framework for Spatiotemporal Super-resolution of Scientific Simulations](https://mlsanity.com/paper/2311.02328/) (2023)
- [Taper-based scattering formulation of the Helmholtz equation to improve the training process of Physics-Informed Neural Networks](https://mlsanity.com/paper/2404.09794/) (2024)
- [Deep regularization networks for inverse problems with noisy operators](https://mlsanity.com/paper/2506.07008/) (2025)

## Abstract (arXiv)

Adaptive loss function formulation is an active area of research and has gained a great deal of popularity in recent years, following the success of deep learning. However, existing frameworks of adaptive loss functions often suffer from slow convergence and poor choice of weights for the loss components. Traditionally, the elements of a multi-part loss function are weighted equally or their weights are determined through heuristic approaches that yield near-optimal (or sub-optimal) results. To address this problem, we propose a family of methods, called SoftAdapt, that dynamically change function weights for multi-part loss functions based on live performance statistics of the component losses. SoftAdapt is mathematically intuitive, computationally efficient and straightforward to implement. In this paper, we present the mathematical formulation and pseudocode for SoftAdapt, along with results from applying our methods to image reconstruction (Sparse Autoencoders) and synthetic data generation (Introspective Variational Autoencoders).