# Fault-tolerant foundation models

arXiv: https://arxiv.org/abs/2610.10311 · HTML: https://mlsanity.com/paper/2610.10311/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 7 of its 64 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 7 of 7)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Chinchilla Scaling: A replication attempt](https://mlsanity.com/paper/2404.10102/) (2024)
- [The Super Weight in Large Language Models](https://mlsanity.com/paper/2411.07191/) (2024)
- [Hidden Dynamics of Massive Activations in Transformer Training](https://mlsanity.com/paper/2508.03616/) (2025)

## Abstract (arXiv)

Emerging computer hardware often trades reliability for energy efficiency; here we show that large-language models (LLMs) can be trained to tolerate this unreliability, and that rather than degrading, their error resilience actually increases as they grow. Modified neural scaling laws inferred from 40,000 GPU-hours of training runs on simulated faulty digital hardware quantify this trend and suggest that models learn to compute within "good" error-correcting codes, whose relative overhead remains finite no matter how large the model gets. This finding leads us to conjecture that appropriately trained LLMs may be formally fault-tolerant; if true, running AI inference on low energy, faulty hardware may be a path to substantial energy savings over the status quo.