# SIGMA: Scalable Spectral Insights for LLM Model Collapse

arXiv: https://arxiv.org/abs/2601.03385 · HTML: https://mlsanity.com/paper/2601.03385/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Model collapse from synthetic data](https://mlsanity.com/synth/model-collapse-from-synthetic-data/)

## Builds on (most cited first, 10 of 10)

- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse](https://mlsanity.com/paper/2404.05090/) (2024)
- [Strong Model Collapse](https://mlsanity.com/paper/2410.04840/) (2024)
- [How to Synthesize Text Data without Model Collapse?](https://mlsanity.com/paper/2412.14689/) (2024)
- [LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs](https://mlsanity.com/paper/2506.15690/) (2025)
- [Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty](https://mlsanity.com/paper/2509.23002/) (2025)
- [Taming Variability: Randomized and Bootstrapped Conformal Risk Control for LLMs](https://mlsanity.com/paper/2509.23007/) (2025)

## Abstract (arXiv)

The rapid adoption of synthetic data for training Large Language Models (LLMs) has introduced the technical challenge of "model collapse"-a degenerative process where recursive training on model-generated content leads to a contraction of distributional variance and representational quality. While the phenomenology of collapse is increasingly evident, rigorous methods to quantify and predict its onset in high-dimensional spaces remain elusive. In this paper, we introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework that benchmarks model collapse through the spectral lens of the embedding Gram matrix. By deriving and utilizing deterministic and stochastic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space. Crucially, our stochastic formulation enables scalable estimation of these bounds, making the framework applicable to large-scale foundation models where full eigendecomposition is intractable. We demonstrate that SIGMA effectively captures the transition towards degenerate states, offering both theoretical insights into the mechanics of collapse and a practical, scalable tool for monitoring the health of recursive training pipelines.