# Evaluating Deep Learning in SystemML using Layer-wise Adaptive Rate Scaling(LARS) Optimizer

arXiv: https://arxiv.org/abs/2102.03018 · HTML: https://mlsanity.com/paper/2102.03018/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 5 Feb 2021.

## Builds on (most cited first, 6 of 6)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [MLlib: Machine Learning in Apache Spark](https://mlsanity.com/paper/1505.06807/) (2015)
- [Adaptive Weight Decay for Deep Neural Networks](https://mlsanity.com/paper/1907.08931/) (2019)
- [Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learning](https://mlsanity.com/paper/1904.11121/) (2019)

## Built on by (most cited first, 1 of 1)

- [Self-supervised learning for radio-astronomy source classification: a benchmark](https://mlsanity.com/paper/2411.14078/) (2024)

## Abstract (arXiv)

Increasing the batch size of a deep learning model is a challenging task. Although it might help in utilizing full available system memory during training phase of a model, it results in significant loss of test accuracy most often. LARS solved this issue by introducing an adaptive learning rate for each layer of a deep learning model. However, there are doubts on how popular distributed machine learning systems such as SystemML or MLlib will perform with this optimizer. In this work, we apply LARS optimizer to a deep learning model implemented using SystemML.We perform experiments with various batch sizes and compare the performance of LARS optimizer with textitStochastic Gradient Descent. Our experimental results show that LARS optimizer performs significantly better than Stochastic Gradient Descent for large batch sizes even with the distributed machine learning framework, SystemML.