# Training Overhead Ratio: A Practical Reliability Metric for Large Language Model Training Systems

arXiv: https://arxiv.org/abs/2408.07482 · HTML: https://mlsanity.com/paper/2408.07482/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 6 references that are in the map.
- First posted on arXiv 14 Aug 2024.

## Builds on (most cited first, 4 of 4)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)

## Built on by (most cited first, 2 of 2)

- [Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets](https://mlsanity.com/paper/2505.12038/) (2025)
- [Hardware-Aware DNN Compression for Homogeneous Edge Devices](https://mlsanity.com/paper/2501.15240/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are revolutionizing the AI industry with their superior capabilities. Training these models requires large-scale GPU clusters and significant computing time, leading to frequent failures that significantly increase training costs. Despite its significance, this field lacks a metric for evaluating reliability. In this work, we introduce a novel reliability metric called Training Overhead Ratio (TOR) to evaluate the reliability of fault-tolerant LLM training systems. TOR is defined as the ratio of optimal training time to the observed training time of a system, serving as a practical tool for users to estimate the actual time required to train an LLM on a given system. Furthermore, our investigation identifies the key factor for enhancing reliability and present TOR equations for various types of failures encountered in practice.