# Auto-Differentiation of Relational Computations for Very Large Scale Machine Learning

arXiv: https://arxiv.org/abs/2306.00088 · HTML: https://mlsanity.com/paper/2306.00088/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 94 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 30 of 45)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://mlsanity.com/paper/1512.01274/) (2015)
- [Beyond Data and Model Parallelism for Deep Neural Networks](https://mlsanity.com/paper/1807.05358/) (2018)
- [Mesh-TensorFlow: Deep Learning for Supercomputers](https://mlsanity.com/paper/1811.02084/) (2018)
- [AliGraph: A Comprehensive Graph Neural Network Platform](https://mlsanity.com/paper/1902.08730/) (2019)
- [DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs](https://mlsanity.com/paper/2010.05337/) (2020)
- [Towards Deeper Graph Neural Networks with Differentiable Group Normalization](https://mlsanity.com/paper/2006.06972/) (2020)
- [GSPMD: General and Scalable Parallelization for ML Computation Graphs](https://mlsanity.com/paper/2105.04663/) (2021)
- [A Differentiable Physics Engine for Deep Learning in Robotics](https://mlsanity.com/paper/1611.01652/) (2016)
- [NeuralSim: Augmenting Differentiable Simulators with Neural Networks](https://mlsanity.com/paper/2011.04217/) (2020)
- [Pathways: Asynchronous Distributed Dataflow for ML](https://mlsanity.com/paper/2203.12533/) (2022)
- [DGL-KE: Training Knowledge Graph Embeddings at Scale](https://mlsanity.com/paper/2004.08532/) (2020)
- [A mathematical model for automatic differentiation in machine learning](https://mlsanity.com/paper/2006.02080/) (2020)
- [PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication](https://mlsanity.com/paper/2203.10428/) (2022)
- [BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling](https://mlsanity.com/paper/2203.10983/) (2022)
- [Automatic Differentiation of Programs with Discrete Randomness](https://mlsanity.com/paper/2210.08572/) (2022)
- [Automatic differentiation of nonsmooth iterative algorithms](https://mlsanity.com/paper/2206.00457/) (2022)
- [Super-efficiency of automatic differentiation for functions defined as a minimum](https://mlsanity.com/paper/2002.03722/) (2020)
- [Randomized Automatic Differentiation](https://mlsanity.com/paper/2007.10412/) (2020)
- [Chainer: A Deep Learning Framework for Accelerating the Research Cycle](https://mlsanity.com/paper/1908.00213/) (2019)
- [Automatic differentiation in ML: Where we are and where we should be going](https://mlsanity.com/paper/1810.11530/) (2018)
- [On Correctness of Automatic Differentiation for Non-Differentiable Functions](https://mlsanity.com/paper/2006.06903/) (2020)
- [End-to-end Optimization of Machine Learning Prediction Queries](https://mlsanity.com/paper/2206.00136/) (2022)
- [Serving Deep Learning Models with Deduplication from Relational Databases](https://mlsanity.com/paper/2201.10442/) (2022)
- [Query Processing on Tensor Computation Runtimes](https://mlsanity.com/paper/2203.01877/) (2022)

## Built on by (most cited first, 1 of 1)

- [The Duck's Brain: Training and Inference of Neural Networks in Modern Database Engines](https://mlsanity.com/paper/2312.17355/) (2023)

## Abstract (arXiv)

The relational data model was designed to facilitate large-scale data management and analytics. We consider the problem of how to differentiate computations expressed relationally. We show experimentally that a relational engine running an auto-differentiated relational algorithm can easily scale to very large datasets, and is competitive with state-of-the-art, special-purpose systems for large-scale distributed machine learning.