# JUWELS Booster -- A Supercomputer for Large-Scale AI Research

arXiv: https://arxiv.org/abs/2108.11976 · HTML: https://mlsanity.com/paper/2108.11976/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 71 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [ZeRO-Offload: Democratizing Billion-Scale Model Training](https://mlsanity.com/paper/2101.06840/) (2021)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Horovod: fast and easy distributed deep learning in TensorFlow](https://mlsanity.com/paper/1802.05799/) (2018)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization](https://mlsanity.com/paper/1905.13727/) (2019)
- [BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding](https://mlsanity.com/paper/1902.06148/) (2019)
- [Measuring the Effects of Data Parallelism on Neural Network Training](https://mlsanity.com/paper/1811.03600/) (2018)
- [Hierarchical Representations for Efficient Architecture Search](https://mlsanity.com/paper/1711.00436/) (2017)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks](https://mlsanity.com/paper/1511.06314/) (2015)
- [COVID-19 Image Data Collection: Prospective Predictions Are the Future](https://mlsanity.com/paper/2006.11988/) (2020)
- [Deep learning with Elastic Averaging SGD](https://mlsanity.com/paper/1412.6651/) (2014)
- [Self-supervised learning through the eyes of a child](https://mlsanity.com/paper/2007.16189/) (2020)
- [8-Bit Approximations for Parallelism in Deep Learning](https://mlsanity.com/paper/1511.04561/) (2015)
- [The AI Index 2021 Annual Report](https://mlsanity.com/paper/2103.06312/) (2021)
- [Exascale Deep Learning for Climate Analytics](https://mlsanity.com/paper/1810.01993/) (2018)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)
- [BigEarthNet Dataset with A New Class-Nomenclature for Remote Sensing Image Understanding](https://mlsanity.com/paper/2001.06372/) (2020)

## Built on by (most cited first, 10 of 10)

- [Lossy Neural Compression for Geospatial Analytics: A Review](https://mlsanity.com/paper/2503.01505/) (2025)
- [Parameter-efficient Bayesian Neural Networks for Uncertainty-aware Depth Estimation](https://mlsanity.com/paper/2409.17085/) (2024)
- [OneProt: Towards Multi-Modal Protein Foundation Models](https://mlsanity.com/paper/2411.04863/) (2024)
- [Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks](https://mlsanity.com/paper/2501.12824/) (2025)
- [1LoRA: Summation Compression for Very Low-Rank Adaptation](https://mlsanity.com/paper/2503.08333/) (2025)
- [Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning](https://mlsanity.com/paper/2505.14125/) (2025)
- [Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs](https://mlsanity.com/paper/2602.03493/) (2026)
- [Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models](https://mlsanity.com/paper/2604.12391/) (2026)
- [Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems](https://mlsanity.com/paper/2412.02729/) (2024)
- [Evolutionary Architecture Search through Grammar-Based Sequence Alignment](https://mlsanity.com/paper/2512.04992/) (2025)

## Abstract (arXiv)

In this article, we present JUWELS Booster, a recently commissioned high-performance computing system at the J"ulich Supercomputing Center. With its system architecture, most importantly its large number of powerful Graphics Processing Units (GPUs) and its fast interconnect via InfiniBand, it is an ideal machine for large-scale Artificial Intelligence (AI) research and applications. We detail its system architecture, parallel, distributed model training, and benchmarks indicating its outstanding performance. We exemplify its potential for research application by presenting large-scale AI research highlights from various scientific fields that require such a facility.