mlsanityData to 7 Oct 2026

On Training Derivative-Constrained Neural Networks

arXiv 2310.01649 ↗First posted 2 Oct 2023cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We refer to the setting where the (partial) derivatives of a neural network's (NN's) predictions with respect to its inputs are used as additional training signal as a derivative-constrained (DC) NN. This situation is common in physics-informed settings in the natural sciences. We propose an integrated RELU (IReLU) activation function to improve training of DC NNs. We also investigate denormalization and label rescaling to help stabilize DC training. We evaluate our methods on physics-informed settings including quantum chemistry and Scientific Machine Learning (SciML) tasks. We demonstrate that existing architectures with IReLU activations combined with denormalization and label rescaling better incorporate training signal provided by derivative constraints.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 48 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

26 of 48 references.

References · 26

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  3. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  5. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  6. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  7. 2019DeepXDE: A deep learning library for solving differential equationscited by 617
  8. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 569
  9. 2020Directional Message Passing for Molecular Graphscited by 556
  10. 2021Equivariant message passing for the prediction of tensorial properties and molecular spectracited by 438
  11. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
  12. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 342
14 more
  1. 2020Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Moleculescited by 254
  2. 2020The Open Catalyst 2020 (OC20) Dataset and Community Challengescited by 246
  3. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 201
  4. 2019Deep neural network solution of the electronic Schr"odinger equationcited by 124
  5. 2023Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNscited by 113
  6. 2021Multi-Objective Loss Balancing for Physics-Informed Deep Learningcited by 86
  7. 2020Physics-informed neural network for ultrasound nondestructive quantification of surface breaking crackscited by 67
  8. 2022When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learningcited by 64
  9. 2022GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasetscited by 61
  10. 2021ForceNet: A Graph Neural Network for Large-Scale Quantum Calculationscited by 58
  11. 2022Spherical Channels for Modeling Atomic Interactionscited by 47
  12. 2019SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functionscited by 38
  13. 2022Ab-initio quantum chemistry with neural-network wavefunctionscited by 28
  14. 2020MTAdam: Automatic Balancing of Multiple Training Loss Termscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Physics-Informed Neural Networks for Solving Derivative-Constrained PDEscited by 0