# Tensor-Parallelism with Partially Synchronized Activations

arXiv: https://arxiv.org/abs/2506.19645 · HTML: https://mlsanity.com/paper/2506.19645/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 24 Jun 2025.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads](https://mlsanity.com/paper/2105.05720/) (2021)
- [Context Parallelism for Scalable Million-Token Inference](https://mlsanity.com/paper/2411.01783/) (2024)
- [At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?](https://mlsanity.com/paper/1909.12340/) (2019)
- [Does compressing activations help model parallel training?](https://mlsanity.com/paper/2301.02654/) (2023)
- [Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference](https://mlsanity.com/paper/2412.04964/) (2024)
- [Domino: Eliminating Communication in LLM Training via Generic Tensor Slicing and Overlapping](https://mlsanity.com/paper/2409.15241/) (2024)
- [Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping](https://mlsanity.com/paper/2501.06589/) (2025)
- [Computation vs. Communication Scaling for Future Transformers on Future Hardware](https://mlsanity.com/paper/2302.02825/) (2023)
- [Activations and Gradients Compression for Model-Parallel Training](https://mlsanity.com/paper/2401.07788/) (2024)
- [SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models](https://mlsanity.com/paper/2502.20727/) (2025)

## Built on by (most cited first, 4 of 4)

- [TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training](https://mlsanity.com/paper/2604.24088/) (2026)
- [TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference](https://mlsanity.com/paper/2508.15881/) (2025)
- [FarSkip-Collective: Unhobbling Blocking Communication in Mixture of Experts Models](https://mlsanity.com/paper/2511.11505/) (2025)
- [Affinity-Aware Sharding for Delayed Tensor Parallelism](https://mlsanity.com/paper/2609.13846/) (2026)

## Abstract (arXiv)

Training and inference of Large Language Models (LLMs) with tensor-parallelism requires substantial communication to synchronize activations. Our findings suggest that with a few minor adjustments to current practices, LLMs can be trained without fully synchronizing activations, reducing bandwidth demands. We name this "Communication-Aware Architecture for Tensor-parallelism" (CAAT-Net). We train a 7B parameter CAAT-Net model and show that tensor-parallel communication can be reduced by up to 50% with no significant drop in pretraining accuracy across nearly all evaluated benchmarks. We also experiment with smaller 130M and 1.1B models to show the robustness and scalability of our method. We find that, in some scenarios, validation loss can even improve when reducing communication. Finally, we demonstrate how CAAT-Net accelerates both training and inference workloads across various settings and model sizes.