mlsanityData to 14 Oct 2026

Unifying back-propagation and forward-forward algorithms through model predictive control

arXiv 2409.19561 ↗First posted 29 Sep 2024cs.LG math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce a Model Predictive Control (MPC) framework for training deep neural networks, systematically unifying the Back-Propagation (BP) and Forward-Forward (FF) algorithms. At the same time, it gives rise to a range of intermediate training algorithms with varying look-forward horizons, leading to a performance-efficiency trade-off. We perform a precise analysis of this trade-off on a deep linear network, where the qualitative conclusions carry over to general networks. Based on our analysis, we propose a principled method to choose the optimization horizon based on given objectives and model specifications. Numerical results on various models and tasks demonstrate the versatility of our method.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

23 of 29 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  7. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  8. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
  9. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  10. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  11. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 137
  12. 2019Training Neural Networks with Local Error Signalscited by 136
11 more
  1. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 115
  2. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  3. 2017Maximum Principle Based Algorithms for Deep Learningcited by 78
  4. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  5. 2020LoCo: Local Contrastive Representation Learningcited by 55
  6. 2019Adaptively Truncating Backpropagation Through Time to Control Gradient Biascited by 22
  7. 2018Stochastic Gradient Descent with Biased but Consistent Gradient Estimatorscited by 18
  8. 2024PIDformer: Transformer Meets Control Theorycited by 11
  9. 2020Learning Quantized Neural Nets by Coarse Gradient Method for Non-linear Classificationcited by 8
  10. 2022Deep Linear Networks for Matrix Completion -- An Infinite Depth Limitcited by 8
  11. 2022Empowering Optimal Control with Machine Learning: A Perspective from Model Predictive Controlcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Learning without Global Backpropagation via Synergistic Information Distillationcited by 1
  2. 2025Forward-Cooperation-Backward (FCB) learning in a Multi-Encoding Uni-Decoding neural network architecturecited by 0
  3. 2026A First-Principles Theory of Slow Thinking and Active Perceptioncited by 0