# BigDL: A Distributed Deep Learning Framework for Big Data

arXiv: https://arxiv.org/abs/1804.05839 · HTML: https://mlsanity.com/paper/1804.05839/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2018.

## Built on by (most cited first, 4 of 4)

- [Machine Learning Systems: A Survey from a Data-Oriented Perspective](https://mlsanity.com/paper/2302.04810/) (2023)
- [A Transistor Operations Model for Deep Learning Energy Consumption Scaling Law](https://mlsanity.com/paper/2205.15062/) (2022)
- [Benchmark Assessment for DeepSpeed Optimization Library](https://mlsanity.com/paper/2202.12831/) (2022)
- [BigDL 2.0: Seamless Scaling of AI Pipelines from Laptops to Distributed Cluster](https://mlsanity.com/paper/2204.01715/) (2022)

## Abstract (arXiv)

This paper presents BigDL (a distributed deep learning framework for Apache Spark), which has been used by a variety of users in the industry for building deep learning applications on production big data platforms. It allows deep learning applications to run on the Apache Hadoop/Spark cluster so as to directly process the production data, and as a part of the end-to-end data analysis pipeline for deployment and management. Unlike existing deep learning frameworks, BigDL implements distributed, data parallel training directly on top of the functional compute model (with copy-on-write and coarse-grained operations) of Spark. We also share real-world experience and "war stories" of users that have adopted BigDL to address their challenges(i.e., how to easily build end-to-end data analysis and deep learning pipelines for their production data).