# The MIT Supercloud Dataset

arXiv: https://arxiv.org/abs/2108.02037 · HTML: https://mlsanity.com/paper/2108.02037/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 29 references that are in the map.
- First posted on arXiv 4 Aug 2021.

## Builds on (most cited first, 4 of 4)

- [Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis](https://mlsanity.com/paper/1807.07814/) (2018)
- [Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads](https://mlsanity.com/paper/1901.05758/) (2019)
- [Convergence of Artificial Intelligence and High Performance Computing on NSF-supported Cyberinfrastructure](https://mlsanity.com/paper/2003.08394/) (2020)
- [Deploying AI Frameworks on Secure HPC Systems with Containers](https://mlsanity.com/paper/1905.10090/) (2019)

## Built on by (most cited first, 7 of 7)

- [Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models](https://mlsanity.com/paper/2205.09646/) (2022)
- [The Unseen AI Disruptions for Power Grids: LLM-Induced Transients](https://mlsanity.com/paper/2409.11416/) (2024)
- [The MIT Supercloud Workload Classification Challenge](https://mlsanity.com/paper/2204.05839/) (2022)
- [Developing a Series of AI Challenges for the United States Department of the Air Force](https://mlsanity.com/paper/2207.07033/) (2022)
- [Improving AI Efficiency in Data Centres by Power Dynamic Response](https://mlsanity.com/paper/2510.11119/) (2025)
- [An Evaluation of Low Overhead Time Series Preprocessing Techniques for Downstream Machine Learning](https://mlsanity.com/paper/2209.05300/) (2022)
- [Dynamic Load Model for Data Centers with Pattern-Consistent Calibration](https://mlsanity.com/paper/2602.07859/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) and Machine learning (ML) workloads are an increasingly larger share of the compute workloads in traditional High-Performance Computing (HPC) centers and commercial cloud systems. This has led to changes in deployment approaches of HPC clusters and the commercial cloud, as well as a new focus on approaches to optimized resource usage, allocations and deployment of new AI frame- works, and capabilities such as Jupyter notebooks to enable rapid prototyping and deployment. With these changes, there is a need to better understand cluster/datacenter operations with the goal of developing improved scheduling policies, identifying inefficiencies in resource utilization, energy/power consumption, failure prediction, and identifying policy violations. In this paper we introduce the MIT Supercloud Dataset which aims to foster innovative AI/ML approaches to the analysis of large scale HPC and datacenter/cloud operations. We provide detailed monitoring logs from the MIT Supercloud system, which include CPU and GPU usage by jobs, memory usage, file system logs, and physical monitoring data. This paper discusses the details of the dataset, collection methodology, data availability, and discusses potential challenge problems being developed using this data. Datasets and future challenge announcements will be available via https://dcc.mit.edu.