mlsanityData to 7 Oct 2026

Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networks

arXiv 2607.13380 ↗First posted 15 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Predictive Coding (PC) offers a biologically motivated alternative to backpropagation via local weight updates, yet routing error between layers still relies on an autograd Jacobian-transpose (J^top) product - the last non-local operation in PC. We show that this dependency is largely avoidable. For any layer f(x)=Act(Norm(L(x))) with frozen normalization statistics, the exact J^top factors into three locally available terms, J^top v = L^top(s odot σ'(z) odot v), where σ' is the activation derivative, z is the pre-activation, and s=γ/σrun is the normalization gain. Prior weight-feedback methods omitted both corrections; restoring them closes the transport gap for this layer class. Locality here holds up to three assumptions, which we state upfront: weight symmetry (L^top mirrors the forward operator, as assumed by all PC), a soft spectral-norm control that is not synapse-local, and a nearest-neighbour approximation for MaxPool. Substituting the identity into PC yields WF-Act-PC, which removes the autograd backward pass from error transport. On CIFAR-10/100 (50 epochs, 5 seeds), WF-Act-PC is the only PC method whose accuracy improves with depth, surpassing iPC - the strongest classical PC baseline - by 2.7-22.3 pp on CIFAR-10. With both methods tuned per architecture, it matches or exceeds a comparably-tuned backpropagation baseline on the deeper CIFAR-10 architectures (VGG-9: 93.57% vs. 92.43%; ResNet-18: 92.76% vs. 91.54%) and on the harder Tiny-ImageNet benchmark, while trailing tuned BP on the deeper CIFAR-100 VGG cells. Our WF-Act-PC implementation is publicly available at https://github.com/jlshen025/pcax

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 28 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

18 of 28 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2018Group Normalizationcited by 1,514
  8. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  9. 2014Difference Target Propagationcited by 144
  10. 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 100
  11. 2020Predictive Coding Approximates Backprop along Arbitrary Computation Graphscited by 89
  12. 2019Deep Learning without Weight Transportcited by 87
6 more
  1. 2020Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architecturescited by 56
  2. 2024Benchmarking Predictive Coding Networks -- Made Simplecited by 29
  3. 2022A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networkscited by 21
  4. 2020Two Routes to Scalable Credit Assignment without Weight Symmetrycited by 19
  5. 2025μPC: Scaling Predictive Coding to 100+ Layer Networkscited by 14
  6. 2025ePC: Fast and Deep Predictive Coding in Digital Simulationcited by 4

Built on by

Not cited yet.