# Scaling Laws Beyond Backpropagation

arXiv: https://arxiv.org/abs/2210.14593 · HTML: https://mlsanity.com/paper/2210.14593/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Reducing Activation Recomputation in Large Transformer Models](https://mlsanity.com/paper/2205.05198/) (2022)
- [Unified Scaling Laws for Routed Language Models](https://mlsanity.com/paper/2202.01169/) (2022)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?](https://mlsanity.com/paper/2204.05832/) (2022)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Revisiting Neural Scaling Laws in Language and Vision](https://mlsanity.com/paper/2209.06640/) (2022)
- [Greedy Layerwise Learning Can Scale to ImageNet](https://mlsanity.com/paper/1812.11446/) (2018)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures](https://mlsanity.com/paper/1807.04587/) (2018)
- [Do Transformer Modifications Transfer Across Implementations and Applications?](https://mlsanity.com/paper/2102.11972/) (2021)
- [Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures](https://mlsanity.com/paper/2006.12878/) (2020)
- [Data Scaling Laws in NMT: The Effect of Noise and Architecture](https://mlsanity.com/paper/2202.01994/) (2022)
- [Principled Training of Neural Networks with Direct Feedback Alignment](https://mlsanity.com/paper/1906.04554/) (2019)
- [Parallel Training of Deep Networks with Local Updates](https://mlsanity.com/paper/2012.03837/) (2020)
- [Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment](https://mlsanity.com/paper/2111.06862/) (2021)
- [Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment](https://mlsanity.com/paper/2012.06373/) (2020)
- [Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms](https://mlsanity.com/paper/2108.13446/) (2021)
- [Differentially Private Deep Learning with Direct Feedback Alignment](https://mlsanity.com/paper/2010.03701/) (2020)
- [Adversarial Robustness by Design through Analog Computing and Synthetic Gradients](https://mlsanity.com/paper/2101.02115/) (2021)
- [Photonic Differential Privacy with Direct Feedback Alignment](https://mlsanity.com/paper/2106.03645/) (2021)

## Built on by (most cited first, 4 of 4)

- [(Mis)Fitting: A Survey of Scaling Laws](https://mlsanity.com/paper/2502.18969/) (2025)
- [Model Merging Scaling Laws in Large Language Models](https://mlsanity.com/paper/2509.24244/) (2025)
- [SAL: Selective Adaptive Learning for Backpropagation-Free Training with Sparsification](https://mlsanity.com/paper/2601.21561/) (2026)
- [Can Local Learning Match Self-Supervised Backpropagation?](https://mlsanity.com/paper/2601.21683/) (2026)

## Abstract (arXiv)

Alternatives to backpropagation have long been studied to better understand how biological brains may learn. Recently, they have also garnered interest as a way to train neural networks more efficiently. By relaxing constraints inherent to backpropagation (e.g., symmetric feedforward and feedback weights, sequential updates), these methods enable promising prospects, such as local learning. However, the tradeoffs between different methods in terms of final task performance, convergence speed, and ultimately compute and data requirements are rarely outlined. In this work, we use scaling laws to study the ability of Direct Feedback Alignment~(DFA) to train causal decoder-only Transformers efficiently. Scaling laws provide an overview of the tradeoffs implied by a modeling decision, up to extrapolating how it might transfer to increasingly large models. We find that DFA fails to offer more efficient scaling than backpropagation: there is never a regime for which the degradation in loss incurred by using DFA is worth the potential reduction in compute budget. Our finding comes at variance with previous beliefs in the alternative training methods community, and highlights the need for holistic empirical approaches to better understand modeling decisions.