# Understanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thought

arXiv: https://arxiv.org/abs/2402.04004 · HTML: https://mlsanity.com/paper/2402.04004/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 0 of its 42 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Built on by (most cited first, 19 of 19)

- [Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?](https://mlsanity.com/paper/2410.23856/) (2024)
- [A Survey of Small Language Models](https://mlsanity.com/paper/2410.20011/) (2024)
- [Shape of Thought: When Distribution Matters More than Correctness in Reasoning Tasks](https://mlsanity.com/paper/2512.22255/) (2025)
- [Enhancing LLM Steering through Sparse Autoencoder-Based Vector Refinement](https://mlsanity.com/paper/2509.23799/) (2025)
- [On Code-Induced Reasoning in LLMs](https://mlsanity.com/paper/2509.21499/) (2025)
- [Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models](https://mlsanity.com/paper/2407.02716/) (2024)
- [Watch Your Steps: Observable and Modular Chains of Thought](https://mlsanity.com/paper/2409.15359/) (2024)
- [3DS: Medical Domain Adaptation of LLMs via Decomposed Difficulty-based Data Selection](https://mlsanity.com/paper/2410.10901/) (2024)
- [Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?](https://mlsanity.com/paper/2407.00996/) (2024)
- [DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training](https://mlsanity.com/paper/2512.03847/) (2025)
- [An Alternative Trajectory for Generative AI](https://mlsanity.com/paper/2603.14147/) (2026)
- [Do Safety Guardrails Need to Reason? LeanGuard: A Fast and Light Approach for Robust Moderation](https://mlsanity.com/paper/2606.26686/) (2026)
- [Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation](https://mlsanity.com/paper/2505.19529/) (2025)
- [An Empirical Study of Sample Selection Strategies for Large Language Model Repair](https://mlsanity.com/paper/2510.20428/) (2025)
- [CoGraM: Context-sensitive granular optimization method with rollback for robust model fusion](https://mlsanity.com/paper/2512.03610/) (2025)
- [SafeLens: Deliberate and Efficient Video Guardrails with Fast-and-Slow Screening](https://mlsanity.com/paper/2605.17610/) (2026)
- [Learning to Adapt SFT Data for Better Reasoning Generalization](https://mlsanity.com/paper/2605.26924/) (2026)
- [Quantifying Error Tolerance in Synthetic Data: An Atomic-level Operand vs. Operator Perturbation Study](https://mlsanity.com/paper/2608.29144/) (2026)
- [Decoupling Token Roles in Autoregressive Pretraining](https://mlsanity.com/paper/2609.33405/) (2026)

## Abstract (arXiv)

During both pretraining and fine-tuning, Large Language Models (LLMs) are trained on trillions of tokens of text of widely varying quality. Both phases of training typically involve heuristically filtering out ``low-quality'' or textitnoisy training samples, yet little is known quantitatively about how the type or intensity of noise affects downstream performance. In this work, we study how noise in chain of thought (CoT) impacts task performance in the highly-controlled setting of algorithmically solvable tasks. First, we develop the Traced Integer (TInt) framework to generate highly customizable noised execution traces for any arithmetic function on lists of integers. We then define two types of noise: textitstatic noise, a local form of noise which is applied after the CoT trace is computed, and textitdynamic noise, a global form of noise which propagates errors in the trace as it is computed. We then evaluate the test performance of pretrained models both prompted and fine-tuned on noised datasets with varying levels of dataset contamination and intensity. We find fine-tuned models are extremely robust to high levels of static noise but struggle significantly more with lower levels of dynamic noise. In contrast, few-shot prompted models appear more sensitive to even static noise. We conclude with a discussion of how our findings impact noise filtering best-practices, in particular emphasizing the importance of removing samples containing destructive dynamic noise with global errors.