# SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelism

arXiv: https://arxiv.org/abs/2602.00062 · HTML: https://mlsanity.com/paper/2602.00062/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Builds on (most cited first, 16 of 16)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [Measuring the Effects of Data Parallelism on Neural Network Training](https://mlsanity.com/paper/1811.03600/) (2018)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://mlsanity.com/paper/1905.11786/) (2019)
- [How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation](https://mlsanity.com/paper/1407.7906/) (2014)
- [A Theoretical Framework for Target Propagation](https://mlsanity.com/paper/2006.14331/) (2020)
- [LoCo: Local Contrastive Representation Learning](https://mlsanity.com/paper/2008.01342/) (2020)
- [Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagation](https://mlsanity.com/paper/1906.05560/) (2019)
- [DeInfoReg: A Decoupled Learning Framework for Better Training Throughput](https://mlsanity.com/paper/2506.18193/) (2025)

## Built on by (most cited first, 1 of 1)

- [ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling](https://mlsanity.com/paper/2608.07974/) (2026)

## Abstract (arXiv)

Adopting large-scale AI models in enterprise information systems is often hindered by high training costs and long development cycles, posing a significant managerial challenge. The standard end-to-end backpropagation (BP) algorithm is a primary driver of modern AI, but it is also the source of inefficiency in training deep networks. This paper introduces a new training methodology, Supervised Contrastive Parallel Learning (SCPL), that addresses this issue by decoupling BP and transforming a long gradient flow into multiple short ones. This design enables the simultaneous computation of parameter gradients in different layers, achieving superior model parallelism and enhancing training throughput. Detailed experiments are presented to demonstrate the efficiency and effectiveness of our model compared to BP, Early Exit, GPipe, and Associated Learning (AL), a state-of-the-art method for decoupling backpropagation. By mitigating a fundamental performance bottleneck, SCPL provides a practical pathway for organizations to develop and deploy advanced information systems more cost-effectively and with greater agility. The experimental code is released for reproducibility. https://github.com/minyaho/scpl/