mlsanityData to 14 Oct 2026

SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions

arXiv 1912.12355 ↗First posted 27 Dec 2019cs.LG math.OC stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

Adaptive loss function formulation is an active area of research and has gained a great deal of popularity in recent years, following the success of deep learning. However, existing frameworks of adaptive loss functions often suffer from slow convergence and poor choice of weights for the loss components. Traditionally, the elements of a multi-part loss function are weighted equally or their weights are determined through heuristic approaches that yield near-optimal (or sub-optimal) results. To address this problem, we propose a family of methods, called SoftAdapt, that dynamically change function weights for multi-part loss functions based on live performance statistics of the component losses. SoftAdapt is mathematically intuitive, computationally efficient and straightforward to implement. In this paper, we present the mathematical formulation and pseudocode for SoftAdapt, along with results from applying our methods to image reconstruction (Sparse Autoencoders) and synthetic data generation (Introspective Variational Autoencoders).

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 6; 2024: 7; 2025: 11; 2026: 10.

40 papers.

Sort:

  1. 2021Multi-Objective Loss Balancing for Physics-Informed Deep Learningcited by 87
  2. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  3. 2021DeepGD: A Deep Learning Framework for Graph Drawing Using GNNcited by 10
  4. 2022Preoperative brain tumor imaging: models and software for segmentation and standardized reportingcited by 9
  5. 2025QCPINN: Quantum-Classical Physics-Informed Neural Networks for Solving PDEscited by 9
  6. 2024Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equationscited by 7
  7. 2025Enabling Automatic Differentiation with Mollified Graph Neural Operatorscited by 6
  8. 2023Physics-informed neural networks modeling for systems with moving immersed boundaries: application to an unsteady flow past a plunging foilcited by 4
  9. 2024Domain-decoupled Physics-informed Neural Networks with Closed-form Gradients for Fast Model Learning of Dynamical Systemscited by 4
  10. 2024Physics-Guided Neural Networks for Intraventricular Vector Flow Mappingcited by 3
  11. 2023OWAdapt: An adaptive loss function for deep learning using OWA operatorscited by 2
  12. 2023Understanding the Effect of the Long Tail on Neural Network Compressioncited by 2
28 more
  1. 2024Adaptive Real-Time Multi-Loss Function Optimization Using Dynamic Memory Fusion Framework: A Case Study on Breast Cancer Segmentationcited by 2
  2. 2025Physics-Informed Regression: Parameter Estimation in Parameter-Linear Nonlinear Dynamic Modelscited by 2
  3. 2025SD-RetinaNet: Topologically Constrained Semi-Supervised Retinal Lesion and Layer Segmentation in OCTcited by 2
  4. 2022Applied monocular reconstruction of parametric faces with domain engineeringcited by 1
  5. 2022VI-PINNs: Variance-involved Physics-informed Neural Networks for Fast and Accurate Prediction of Partial Differential Equationscited by 1
  6. 2023On Training Derivative-Constrained Neural Networkscited by 1
  7. 2023DSAF: A Dual-Stage Adaptive Framework for Numerical Weather Prediction Downscalingcited by 1
  8. 2024Network scaling and scale-driven loss balancing for intelligent poroelastographycited by 1
  9. 2025Learning low-dimensional representations of ensemble forecast fields using autoencoder-based methodscited by 1
  10. 2025Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulationcited by 1
  11. 2025Bridging ocean wave physics and deep learning: Physics-informed neural operators for nonlinear wavefield reconstruction in real-timecited by 1
  12. 2025Physics-Informed Neural Operators for Cardiac Electrophysiologycited by 1
  13. 2026Enhancing Physics-Informed Neural Networks with Domain-aware Fourier Features: Towards Improved Performance and Interpretable Resultscited by 1
  14. 2026Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networkscited by 1
  15. 2022Domain Engineering for Applied Monocular Reconstruction of Parametric Facescited by 0
  16. 2023An Operator Learning Framework for Spatiotemporal Super-resolution of Scientific Simulationscited by 0
  17. 2024Taper-based scattering formulation of the Helmholtz equation to improve the training process of Physics-Informed Neural Networkscited by 0
  18. 2025Deep regularization networks for inverse problems with noisy operatorscited by 0
  19. 2025Auto-Adaptive PINNs with Applications to Phase Transitionscited by 0
  20. 2025Learning Low Rank Neural Representations of Hyperbolic Wave Dynamics from Datacited by 0
  21. 2026Informed Machine Learning with Knowledge Landmarkscited by 0
  22. 2026Focus on What Matters: Two-Stage ROI-Aware Refinement for Anatomy-Preserving Fetal Ultrasound Reconstructioncited by 0
  23. 2026Rethinking Loss Reweighting for Imbalance Learning as an Inverse Problem: A Neural Collapse Point of Viewcited by 0
  24. 2026Overcoming the Limits of Finite Difference Method; Physics-Informed Neural Network for Noisy High-Dimensional Heat Diffusioncited by 0
  25. 2026Advances in Scientific Machine Learning for Coupled Fluid Flow and Transportcited by 0
  26. 2026Neptuna: A Comprehensive Machine Learning Framework for Benchmarking Complex Multiphase Flowscited by 0
  27. 2026Semi-Supervised Learning for Molecular Graphs via Ensemble Consensuscited by 0
  28. 2026Enriched Holomorphic Neural Networks for 2D and 3D Laplace Problemscited by 0