mlsanityData to 14 Oct 2026

Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networks

arXiv 2308.09858 ↗First posted 18 Aug 2023cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Backward propagation (BP) is widely used to compute the gradients in neural network training. However, it is hard to implement BP on edge devices due to the lack of hardware and software resources to support automatic differentiation. This has tremendously increased the design complexity and time-to-market of on-device training accelerators. This paper presents a completely BP-free framework that only requires forward propagation to train realistic neural networks. Our technical contributions are three-fold. Firstly, we present a tensor-compressed variance reduction approach to greatly improve the scalability of zeroth-order (ZO) optimization, making it feasible to handle a network size that is beyond the capability of previous ZO approaches. Secondly, we present a hybrid gradient evaluation approach to improve the efficiency of ZO training. Finally, we extend our BP-free training framework to physics-informed neural networks (PINNs) by proposing a sparse-grid approach to estimate the derivatives in the loss function without using BP. Our BP-free training only loses little accuracy on the MNIST dataset compared with standard first-order training. We also demonstrate successful results in training a PINN for solving a 20-dim Hamiltonian-Jacobi-Bellman PDE. This memory-efficient and BP-free approach may serve as a foundation for the near-future on-device training on many resource-constraint platforms (e.g., FPGA, ASIC, micro-controllers, and photonic chips).

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 18 Aug 2023.

Builds on

37 of 59 references.

References · 37

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  4. 2015Automatic differentiation in machine learning: a surveycited by 855
  5. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  6. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  7. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  8. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  9. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
  10. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  11. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
  12. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
25 more
  1. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  2. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  3. 2022On-Device Training Under 256KB Memorycited by 160
  4. 2015An Optimal Algorithm for Bandit and Zero-Order Convex Optimization with Two-Point Feedbackcited by 111
  5. 2021CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Methodcited by 93
  6. 2020DeepReach: A Deep Learning Approach to High-Dimensional Reachabilitycited by 91
  7. 2019ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimizationcited by 76
  8. 2022Gradients without Backpropagationcited by 73
  9. 2019A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimizationcited by 68
  10. 2022Scaling Forward Gradient With Local Lossescited by 65
  11. 2020Learning Certified Control using Contraction Metriccited by 49
  12. 2021A Zeroth-Order Block Coordinate Descent Algorithm for Huge-Scale Black-Box Optimizationcited by 46
  13. 2020A mathematical model for automatic differentiation in machine learningcited by 45
  14. 2016A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Ordercited by 44
  15. 2022How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspectivecited by 39
  16. 2019Bayesian Tensorized Neural Networks with Automatic Rank Selectioncited by 26
  17. 2020Efficient On-Chip Learning for Optical Neural Networks Through Power-Aware Sparse Zeroth-Order Optimizationcited by 24
  18. 2022Learning Physics-Informed Neural Networks without Stacked Back-propagationcited by 23
  19. 2021L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimizationcited by 21
  20. 2020Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determinationcited by 18
  21. 2020A Neural Network Approach Applied to Multi-Agent Optimal Controlcited by 14
  22. 2019A Stochastic Derivative-Free Optimization Method with Importance Sampling: Theory and Learning to Controlcited by 9
  23. 2021On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Methodcited by 9
  24. 2023PIFON-EPT: MR-Based Electrical Property Tomography Using Physics-Informed Fourier Networkscited by 4
  25. 20213U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Accelerationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 4; 2026: 1.

12 papers.

  1. 2024Score-Based Physics-Informed Neural Networks for High-Dimensional Fokker-Planck Equationscited by 22
  2. 2024CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimizationcited by 15
  3. 2023Training Convolutional Neural Networks with the Forward-Forward algorithmcited by 14
  4. 2024Separable Operator Networkscited by 5
  5. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  6. 2023Real-Time FJ/MAC PDE Solvers via Tensorized, Back-Propagation-Free Optical PINN Trainingcited by 4
  7. 2025Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimizationcited by 3
  8. 2024Score-fPINN: Fractional Score-Based Physics-Informed Neural Networks for High-Dimensional Fokker-Planck-Levy Equationscited by 1
  9. 2025Low-rank surrogate modeling and stochastic zero-order optimization for training of neural networks with black-box layerscited by 1
  10. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  11. 2025Experimental Demonstration of an Optical Neural PDE Solver via On-Chip PINN Trainingcited by 0
  12. 2025Scaling of hardware-compatible perturbative training algorithmscited by 0