# A Bi-layered Parallel Training Architecture for Large-scale Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1810.07742 · HTML: https://mlsanity.com/paper/1810.07742/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2018.

## Built on by (most cited first, 6 of 6)

- [DAE-PINN: A Physics-Informed Neural Network Model for Simulating Differential-Algebraic Equations with Application to Power Networks](https://mlsanity.com/paper/2109.04304/) (2021)
- [Towards Efficient and Stable K-Asynchronous Federated Learning with Unbounded Stale Gradients on Non-IID Data](https://mlsanity.com/paper/2203.01214/) (2022)
- [Exploring adversarial attacks in federated learning for medical imaging](https://mlsanity.com/paper/2310.06227/) (2023)
- [HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models](https://mlsanity.com/paper/2405.16256/) (2024)
- [VecLSTM: Trajectory Data Processing and Management for Activity Recognition through LSTM Vectorization and Database Integration](https://mlsanity.com/paper/2409.19258/) (2024)
- [Learning to Generate Parameters of ConvNets for Unseen Image Data](https://mlsanity.com/paper/2310.11862/) (2023)

## Abstract (arXiv)

Benefitting from large-scale training datasets and the complex training network, Convolutional Neural Networks (CNNs) are widely applied in various fields with high accuracy. However, the training process of CNNs is very time-consuming, where large amounts of training samples and iterative operations are required to obtain high-quality weight parameters. In this paper, we focus on the time-consuming training process of large-scale CNNs and propose a Bi-layered Parallel Training (BPT-CNN) architecture in distributed computing environments. BPT-CNN consists of two main components: (a) an outer-layer parallel training for multiple CNN subnetworks on separate data subsets, and (b) an inner-layer parallel training for each subnetwork. In the outer-layer parallelism, we address critical issues of distributed and parallel computing, including data communication, synchronization, and workload balance. A heterogeneous-aware Incremental Data Partitioning and Allocation (IDPA) strategy is proposed, where large-scale training datasets are partitioned and allocated to the computing nodes in batches according to their computing power. To minimize the synchronization waiting during the global weight update process, an Asynchronous Global Weight Update (AGWU) strategy is proposed. In the inner-layer parallelism, we further accelerate the training process for each CNN subnetwork on each computer, where computation steps of convolutional layer and the local weight training are parallelized based on task-parallelism. We introduce task decomposition and scheduling strategies with the objectives of thread-level load balancing and minimum waiting time for critical paths. Extensive experimental results indicate that the proposed BPT-CNN effectively improves the training performance of CNNs while maintaining the accuracy.