# Breaking Chains with Trees: Model-Parallel Deep Learning with O(log N) Time Complexity

arXiv: https://arxiv.org/abs/2606.21497 · HTML: https://mlsanity.com/paper/2606.21497/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 19 Jun 2026.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass](https://mlsanity.com/paper/2201.11665/) (2022)
- [A Theoretical Framework for Target Propagation](https://mlsanity.com/paper/2006.14331/) (2020)
- [LoCo: Local Contrastive Representation Learning](https://mlsanity.com/paper/2008.01342/) (2020)
- [Random feedback weights support learning in deep neural networks](https://mlsanity.com/paper/1411.0247/) (2014)
- [Local plasticity rules can learn deep representations using self-supervised contrastive predictions](https://mlsanity.com/paper/2010.08262/) (2020)
- [Towards Scaling Difference Target Propagation by Learning Backprop Targets](https://mlsanity.com/paper/2201.13415/) (2022)
- [Parallel Scheduled Sampling](https://mlsanity.com/paper/1906.04331/) (2019)
- [DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation](https://mlsanity.com/paper/2506.14202/) (2025)
- [Training on the Edge: The why and the how](https://mlsanity.com/paper/1903.03051/) (2019)
- [NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation](https://mlsanity.com/paper/2503.24322/) (2025)
- [Block-local learning with probabilistic latent representations](https://mlsanity.com/paper/2305.14974/) (2023)

## Abstract (arXiv)

Modern deep neural networks are trained using error backpropagation, which requires sequential forward and backward computations across network layers. As these networks become deeper, this introduces limitations, since layer-wise updates are strictly interdependent and cannot proceed in parallel. These constraints restrict training procedures to data-parallel schemes, thereby prohibiting model-parallel training. We propose TreeProp, an architecture-agnostic variational learning framework that organizes network layers into a tree-structured hierarchy. During training, TreeProp replaces sequential forward computations and backward gradient propagation with hierarchical computations. This allows intermediate representations and learning signals to be constructed in time complexity of O(log N) for a network of N layers. To the best of our knowledge, TreeProp is the first learning algorithm for deep neural networks with logarithmic parallel time complexity for both forward computation and backward gradient propagation during training. Furthermore, we show that multiple valid paths through the hierarchy exist, such that TreeProp implicitly learns subnetworks with different effective depths, but without additional training effort. We evaluate TreeProp on vision classification and autoregressive language modeling, matching the performance of conventional end-to-end training for a variety of tasks and outperforming previous contrastive training approaches. We further demonstrate the applicability of TreeProp to recurrent neural networks that otherwise rely on backpropagation through time.