# TorchAO: PyTorch-Native Training-to-Serving Model Optimization

arXiv: https://arxiv.org/abs/2507.16099 · HTML: https://mlsanity.com/paper/2507.16099/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 21 Jul 2025.

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://mlsanity.com/paper/2403.03507/) (2024)
- [Microscaling Data Formats for Deep Learning](https://mlsanity.com/paper/2310.10537/) (2023)
- [TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training](https://mlsanity.com/paper/2410.06511/) (2024)
- [Stochastic Sign Descent Methods: New Algorithms and Better Theory](https://mlsanity.com/paper/1905.12938/) (2019)
- [ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization](https://mlsanity.com/paper/2502.02631/) (2025)
- [Accelerating Transformer Inference and Training with 2:4 Activation Sparsity](https://mlsanity.com/paper/2503.16672/) (2025)

## Built on by (most cited first, 26 of 26)

- [Scaling LLM Test-Time Compute with Mobile NPU on Smartphones](https://mlsanity.com/paper/2509.23324/) (2025)
- [UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs](https://mlsanity.com/paper/2512.03383/) (2025)
- [The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM](https://mlsanity.com/paper/2510.01650/) (2025)
- [StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models](https://mlsanity.com/paper/2604.15416/) (2026)
- [ExecuTorch -- A Unified PyTorch Solution to Run AI Models On-Device](https://mlsanity.com/paper/2605.08195/) (2026)
- [Scalable Neural Decoders for Practical Fault-Tolerant Quantum Computation](https://mlsanity.com/paper/2604.08358/) (2026)
- [Cert-LAS: Toward Certified Model Ownership Verification for Text-to-Image Diffusion Models via Layer-Adaptive Smoothing](https://mlsanity.com/paper/2605.29809/) (2026)
- [Bound to Disagree: Generalization Bounds via Certifiable Surrogates](https://mlsanity.com/paper/2602.23128/) (2026)
- [Zero-Shot Quantization via Weight-Space Arithmetic](https://mlsanity.com/paper/2604.03420/) (2026)
- [AttestLLM: Efficient Attestation Framework for Billion-scale On-device LLMs](https://mlsanity.com/paper/2509.06326/) (2025)
- [LoKA: Low-precision Kernel Applications for Recommendation Models At Scale](https://mlsanity.com/paper/2605.10886/) (2026)
- [GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation](https://mlsanity.com/paper/2606.23419/) (2026)
- [QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs](https://mlsanity.com/paper/2609.39223/) (2026)
- [MXNorm: Reusing MXFP block scales for efficient tensor normalisation](https://mlsanity.com/paper/2603.13180/) (2026)
- [torchtune: PyTorch native post-training library](https://mlsanity.com/paper/2605.21442/) (2026)
- [CAT-Translate: Building Compact Open-Source Models for Japanese-English Translation](https://mlsanity.com/paper/2606.21413/) (2026)
- [Recti-Q: Feature-Space Rectification for Out-of-Distribution-Robust Quantized Perception in Edge Robotics](https://mlsanity.com/paper/2607.18540/) (2026)
- [Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack](https://mlsanity.com/paper/2608.18182/) (2026)
- [A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint](https://mlsanity.com/paper/2608.28003/) (2026)
- [A Generalized Optimization Engine (GOE) for Edge AI Inference Acceleration](https://mlsanity.com/paper/2608.28652/) (2026)
- [Towards Full Pipeline FP8 Reinforcement Learning for LLMs](https://mlsanity.com/paper/2609.22870/) (2026)
- [Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weights](https://mlsanity.com/paper/2609.31564/) (2026)
- [EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitrates](https://mlsanity.com/paper/2609.34185/) (2026)
- [Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference](https://mlsanity.com/paper/2609.36654/) (2026)
- [Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization](https://mlsanity.com/paper/2610.12444/) (2026)
- [Linear-Time Non-Local Codec Placement for Machine Learning Model Compression](https://mlsanity.com/paper/2610.14703/) (2026)

## Abstract (arXiv)

We present TorchAO, a PyTorch-native model optimization framework leveraging quantization and sparsity to provide an end-to-end, training-to-serving workflow for AI models. TorchAO supports a variety of popular model optimization techniques, including FP8 quantized training, quantization-aware training (QAT), post-training quantization (PTQ), and 2:4 sparsity, and leverages a novel tensor subclass abstraction to represent a variety of widely-used, backend agnostic low precision data types, including INT4, INT8, FP8, MXFP4, MXFP6, and MXFP8. TorchAO integrates closely with the broader ecosystem at each step of the model optimization pipeline, from pre-training (TorchTitan) to fine-tuning (TorchTune, Axolotl) to serving (HuggingFace, vLLM, SGLang, ExecuTorch), connecting an otherwise fragmented space in a single, unified workflow. TorchAO has enabled recent launches of the quantized Llama 3.2 1B/3B and LlamaGuard3-8B models and is open-source at https://github.com/pytorch/ao/.