# PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks

arXiv: https://arxiv.org/abs/2210.09147 · HTML: https://mlsanity.com/paper/2210.09147/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 14 of 14)

- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Memory-Efficient Pipeline-Parallel DNN Training](https://mlsanity.com/paper/2006.09503/) (2020)
- [PyTorch Distributed: Experiences on Accelerating Data Parallel Training](https://mlsanity.com/paper/2006.15704/) (2020)
- [torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models](https://mlsanity.com/paper/2004.09910/) (2020)
- [XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training](https://mlsanity.com/paper/1911.04610/) (2019)
- [Deep Constraint-based Propagation in Graph Neural Networks](https://mlsanity.com/paper/2005.02392/) (2020)
- [Multi-GPU Training of ConvNets](https://mlsanity.com/paper/1312.5853/) (2013)
- [Cognitive Action Laws: The Case of Visual Features](https://mlsanity.com/paper/1808.09162/) (2018)
- [Local Propagation in Constraint-based Neural Network](https://mlsanity.com/paper/2002.07720/) (2020)
- [Backprop Diffusion is Biologically Plausible](https://mlsanity.com/paper/1912.04635/) (2019)
- [Learning Visual Features Under Motion Invariance](https://mlsanity.com/paper/1909.00350/) (2019)
- [Learning in Text Streams: Discovery and Disambiguation of Entity and Relation Instances](https://mlsanity.com/paper/1909.05367/) (2019)
- [Developing Constrained Neural Units Over Time](https://mlsanity.com/paper/2009.00296/) (2020)

## Built on by (most cited first, 1 of 1)

- [A Unified Framework for Neural Computation and Learning Over Time](https://mlsanity.com/paper/2409.12038/) (2024)

## Abstract (arXiv)

In this paper, we present PARTIME, a software library written in Python and based on PyTorch, designed specifically to speed up neural networks whenever data is continuously streamed over time, for both learning and inference. Existing libraries are designed to exploit data-level parallelism, assuming that samples are batched, a condition that is not naturally met in applications that are based on streamed data. Differently, PARTIME starts processing each data sample at the time in which it becomes available from the stream. PARTIME wraps the code that implements a feed-forward multi-layer network and it distributes the layer-wise processing among multiple devices, such as Graphics Processing Units (GPUs). Thanks to its pipeline-based computational scheme, PARTIME allows the devices to perform computations in parallel. At inference time this results in scaling capabilities that are theoretically linear with respect to the number of devices. During the learning stage, PARTIME can leverage the non-i.i.d. nature of the streamed data with samples that are smoothly evolving over time for efficient gradient computations. Experiments are performed in order to empirically compare PARTIME with classic non-parallel neural computations in online learning, distributing operations on up to 8 NVIDIA GPUs, showing significant speedups that are almost linear in the number of devices, mitigating the impact of the data transfer overhead.