mlsanityData to 14 Oct 2026

TorchAO: PyTorch-Native Training-to-Serving Model Optimization

arXiv 2507.16099 ↗First posted 21 Jul 2025cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

We present TorchAO, a PyTorch-native model optimization framework leveraging quantization and sparsity to provide an end-to-end, training-to-serving workflow for AI models. TorchAO supports a variety of popular model optimization techniques, including FP8 quantized training, quantization-aware training (QAT), post-training quantization (PTQ), and 2:4 sparsity, and leverages a novel tensor subclass abstraction to represent a variety of widely-used, backend agnostic low precision data types, including INT4, INT8, FP8, MXFP4, MXFP6, and MXFP8. TorchAO integrates closely with the broader ecosystem at each step of the model optimization pipeline, from pre-training (TorchTitan) to fine-tuning (TorchTune, Axolotl) to serving (HuggingFace, vLLM, SGLang, ExecuTorch), connecting an otherwise fragmented space in a single, unified workflow. TorchAO has enabled recent launches of the quantized Llama 3.2 1B/3B and LlamaGuard3-8B models and is open-source at https://github.com/pytorch/ao/.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 17 of its 45 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

17 of 45 references.

References · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
5 more
  1. 2023Microscaling Data Formats for Deep Learningcited by 152
  2. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  3. 2019Stochastic Sign Descent Methods: New Algorithms and Better Theorycited by 49
  4. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  5. 2025Accelerating Transformer Inference and Training with 2:4 Activation Sparsitycited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 22.

26 papers.

Sort:

By area:

  1. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 11
  2. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  3. 2025The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMMcited by 5
  4. 2026StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Modelscited by 5
  5. 2026ExecuTorch -- A Unified PyTorch Solution to Run AI Models On-Devicecited by 5
  6. 2026Scalable Neural Decoders for Practical Fault-Tolerant Quantum Computationcited by 4
  7. 2026Cert-LAS: Toward Certified Model Ownership Verification for Text-to-Image Diffusion Models via Layer-Adaptive Smoothingcited by 3
  8. 2026Bound to Disagree: Generalization Bounds via Certifiable Surrogatescited by 2
  9. 2026Zero-Shot Quantization via Weight-Space Arithmeticcited by 2
  10. 2025AttestLLM: Efficient Attestation Framework for Billion-scale On-device LLMscited by 1
  11. 2026LoKA: Low-precision Kernel Applications for Recommendation Models At Scalecited by 1
  12. 2026GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generationcited by 1
14 more
  1. 2026QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMscited by 1
  2. 2026MXNorm: Reusing MXFP block scales for efficient tensor normalisationcited by 0
  3. 2026torchtune: PyTorch native post-training librarycited by 0
  4. 2026CAT-Translate: Building Compact Open-Source Models for Japanese-English Translationcited by 0
  5. 2026Recti-Q: Feature-Space Rectification for Out-of-Distribution-Robust Quantized Perception in Edge Roboticscited by 0
  6. 2026Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stackcited by 0
  7. 2026A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraintcited by 0
  8. 2026A Generalized Optimization Engine (GOE) for Edge AI Inference Accelerationcited by 0
  9. 2026Towards Full Pipeline FP8 Reinforcement Learning for LLMscited by 0
  10. 2026Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weightscited by 0
  11. 2026EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitratescited by 0
  12. 2026Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inferencecited by 0
  13. 2026Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantizationcited by 0
  14. 2026Linear-Time Non-Local Codec Placement for Machine Learning Model Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing PyTorch Inference with LLM-Based Multi-Agent Systems
  2. 2021BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
  3. 2024LoQT: Low-Rank Adapters for Quantized Pretraining
  4. 2023QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources
  5. 2021Automated Backend-Aware Post-Training Quantization
  6. 2023torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP
  7. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environments
  8. 2025A Selective Quantization Tuner for ONNX Models