mlsanityData to 7 Oct 2026

TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training

arXiv 2604.24088 ↗First posted 27 Apr 2026cs.DC cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Handling communication overhead in large-scale tensor-parallel training remains a critical challenge due to the dense, near-zero distributions of intermediate tensors, which exacerbate errors under frequent communication and introduce significant computational overhead during compression. To this end, we propose TACO (Tensor-parallel Adaptive COmmunication compression), a robust FP8-based framework for compressing TP intermediate tensors. First, we employ a data-driven reshaping strategy combined with an Adaptive Scale-Hadamard Transform to enable high-fidelity FP8 quantization, while its Dual-Scale Quantization mechanism ensures numerical stability throughout training. Second, we design a highly fused compression operator to reduce memory traffic and kernel launch overhead, allowing efficient overlap with communication. Finally, we integrate TACO with existing state-of-the-art methods for Data and Pipeline Parallelism to develop a compression-enabled 3D-parallel training framework. Detailed experiments on GPT models and Qwen model demonstrate up to 1.87X end-to-end throughput improvement while maintaining near-lossless accuracy, validating the effectiveness and efficiency of TACO in large-scale training.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

39 of 55 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  11. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  12. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
27 more
  1. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  2. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  3. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  4. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 125
  5. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  6. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  7. 2022FP8 Quantization: The Power of the Exponentcited by 87
  8. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  9. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  10. 2023QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Modelscited by 39
  11. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 30
  12. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  13. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  14. 2022Near-Optimal Sparse Allreduce for Distributed Deep Learningcited by 23
  15. 2021ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Trainingcited by 22
  16. 2023Quantized Distributed Training of Large Models with Convergence Guaranteescited by 13
  17. 2024SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Trainingcited by 13
  18. 2024Demystifying the Communication Characteristics for Distributed Transformer Modelscited by 7
  19. 2024Activations and Gradients Compression for Model-Parallel Trainingcited by 6
  20. 2024Towards Low-bit Communication for Tensor Parallel LLM Inferencecited by 6
  21. 2023Tango: rethinking quantization for graph neural network training on GPUscited by 4
  22. 2025Tensor-Parallelism with Partially Synchronized Activationscited by 4
  23. 2024Accelerating Large Language Model Training with Hybrid GPU-based Compressioncited by 2
  24. 2025TAGC: Optimizing Gradient Communication in Distributed Transformer Trainingcited by 2
  25. 2025TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Networkcited by 2
  26. 2025EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Trainingcited by 2
  27. 2024Hiding Communication Cost in Distributed LLM Training via Micro-batch Co-executioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Trainingcited by 0