# COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training

arXiv: https://arxiv.org/abs/2410.19313 · HTML: https://mlsanity.com/paper/2410.19313/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 73 references that are in the map.
- First posted on arXiv 25 Oct 2024.

## Builds on (most cited first, 30 of 61)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)

## Built on by (most cited first, 30 of 33)

- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache](https://mlsanity.com/paper/2502.10424/) (2025)
- [Oscillation-Reduced MXFP4 Training for Vision Transformers](https://mlsanity.com/paper/2502.20853/) (2025)
- [Scalable Training of Mixture-of-Experts Models with Megatron Core](https://mlsanity.com/paper/2603.07685/) (2026)
- [HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs](https://mlsanity.com/paper/2501.02625/) (2025)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow](https://mlsanity.com/paper/2601.14243/) (2026)
- [Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention](https://mlsanity.com/paper/2607.04422/) (2026)
- [QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch](https://mlsanity.com/paper/2604.07853/) (2026)
- [Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets](https://mlsanity.com/paper/2603.16731/) (2026)
- [ECO: Quantized Training without Full-Precision Master Weights](https://mlsanity.com/paper/2601.22101/) (2026)
- [FlashOptim: Optimizers for Memory-Efficient Training](https://mlsanity.com/paper/2602.23349/) (2026)
- [StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models](https://mlsanity.com/paper/2604.15416/) (2026)
- [Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design](https://mlsanity.com/paper/2605.15871/) (2026)
- [A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models](https://mlsanity.com/paper/2509.22536/) (2025)
- [ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUs](https://mlsanity.com/paper/2604.03298/) (2026)
- [Stable FP4 Training via Transposition-Invariant Block Quantization](https://mlsanity.com/paper/2607.24953/) (2026)
- [QKV Projections Require a Fraction of Their Memory](https://mlsanity.com/paper/2506.02939/) (2025)
- [MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling](https://mlsanity.com/paper/2511.05811/) (2025)
- [LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs](https://mlsanity.com/paper/2512.15306/) (2025)
- [FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training](https://mlsanity.com/paper/2606.22932/) (2026)
- [MoR: Mixture Of Representations For Mixed-Precision Training](https://mlsanity.com/paper/2512.22804/) (2025)
- [AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation](https://mlsanity.com/paper/2604.02525/) (2026)
- [AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs](https://mlsanity.com/paper/2605.00539/) (2026)
- [GNMR: Runtime Stability Control for Low-Precision Large Language Model Training](https://mlsanity.com/paper/2606.00539/) (2026)
- [GeoFP8: Geometry-Aware FP8 Gradient Compression for Distributed LLM Training](https://mlsanity.com/paper/2607.07494/) (2026)
- [Auditing Invisible Weight Updates with Reference Traces](https://mlsanity.com/paper/2607.09800/) (2026)
- [LaCache: Exact Caching and Precision-Adaptive Inference for Diffusion Large Language Models](https://mlsanity.com/paper/2607.16339/) (2026)
- [Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers](https://mlsanity.com/paper/2608.22322/) (2026)
- [Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs](https://mlsanity.com/paper/2608.26581/) (2026)

## Abstract (arXiv)

FP8 training has emerged as a promising method for improving training efficiency. Existing frameworks accelerate training by applying FP8 computation to linear layers while leaving optimizer states and activations in higher precision, which fails to fully optimize memory usage. This paper introduces COAT (Compressing Optimizer States and Activations for FP8 Training), a novel FP8 training framework designed to significantly reduce memory footprint when training large models. COAT addresses current limitations through two key innovations: (1) Dynamic Range Expansion, which aligns optimizer state distributions more closely with the FP8 representation range, thereby reducing quantization error, and (2) Mixed-Granularity Activation Quantization, which optimizes activation memory using a combination of per-tensor and per-group quantization strategies. Experiments demonstrate that COAT effectively reduces end-to-end training memory footprint by 1.54x compared to BF16 while achieving nearly lossless performance across various tasks, such as Large Language Model pretraining and fine-tuning and Vision Language Model training. COAT also achieves a 1.43x end-to-end training speedup compared to BF16, performing on par with or surpassing TransformerEngine's speedup. COAT enables efficient full-parameter training of large models on fewer GPUs, and facilitates doubling the batch size in distributed training settings, providing a practical solution for scaling large-scale model training. The code is available at https://github.com/NVlabs/COAT.