mlsanityData to 7 Oct 2026

Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient

arXiv 2409.15393 ↗First posted 23 Sep 2024cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural networks (NN) are extensively studied in cutting-edge soft sensor models due to their feature extraction and function approximation capabilities. Current research into network-based methods primarily focuses on models' offline accuracy. Notably, in industrial soft sensor context, online optimizing stability and interpretability are prioritized, followed by accuracy. This requires a clearer understanding of network's training process. To bridge this gap, we propose a novel NN named the Approximated Orthogonal Projection Unit (AOPU) which has solid mathematical basis and presents superior training stability. AOPU truncates the gradient backpropagation at dual parameters, optimizes the trackable parameters updates, and enhances the robustness of training. We further prove that AOPU attains minimum variance estimation (MVE) in NN, wherein the truncated gradient approximates the natural gradient (NG). Empirical results on two chemical process datasets clearly show that AOPU outperforms other models in achieving stable convergence, marking a significant advancement in soft sensor field.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 58 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

26 of 58 references.

References · 26

  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  3. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2014New insights and perspectives on the natural gradient methodcited by 435
  6. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 385
  7. 2013Gaussian Processes for Big Datacited by 377
  8. 2015A Critical Review of Recurrent Neural Networks for Sequence Learningcited by 223
  9. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  10. 2020Efficiently Sampling Functions from Gaussian Process Posteriorscited by 125
  11. 2023OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensemblingcited by 71
  12. 2016Understanding Probabilistic Sparse Gaussian Process Approximationscited by 68
14 more
  1. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  2. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  3. 2017Learning Linear Dynamical Systems via Spectral Filteringcited by 46
  4. 2018Spectral Filtering for General Linear Dynamical Systemscited by 37
  5. 2023Deep networks for system identification: a Surveycited by 35
  6. 2022Random vector functional link network: recent developments, applications, and future directionscited by 33
  7. 2021Optimistic Dual Extrapolation for Coherent Non-monotone Variational Inequalitiescited by 29
  8. 2021Auto-differentiable Ensemble Kalman Filterscited by 28
  9. 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
  10. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  11. 2015Provable Bayesian Inference via Particle Mirror Descentcited by 23
  12. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  13. 2020Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approachcited by 15
  14. 2020Understanding Recurrent Neural Networks Using Nonequilibrium Response Theorycited by 10

Built on by

Not cited yet.