# T3: Transparent Tracking & Triggering for Fine-grained Overlap of Compute & Collectives

arXiv: https://arxiv.org/abs/2401.16677 · HTML: https://mlsanity.com/paper/2401.16677/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 80 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://mlsanity.com/paper/2104.07857/) (2021)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Scalable and Efficient MoE Training for Multitask Multilingual Models](https://mlsanity.com/paper/2109.10465/) (2021)
- [Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads](https://mlsanity.com/paper/2105.05720/) (2021)
- [TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches](https://mlsanity.com/paper/2111.04867/) (2021)
- [Synthesizing Optimal Collective Algorithms](https://mlsanity.com/paper/2008.08708/) (2020)
- [Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms](https://mlsanity.com/paper/2007.00156/) (2020)
- [Analyzing Machine Learning Workloads Using a Detailed GPU Simulator](https://mlsanity.com/paper/1811.08933/) (2018)

## Built on by (most cited first, 16 of 16)

- [MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production](https://mlsanity.com/paper/2505.11432/) (2025)
- [FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models](https://mlsanity.com/paper/2501.10714/) (2025)
- [FlashMoE: Fast Distributed MoE in a Single Kernel](https://mlsanity.com/paper/2506.04667/) (2025)
- [Efficient and Adaptable Overlapping for Computation and Communication via Signaling and Reordering](https://mlsanity.com/paper/2504.19519/) (2025)
- [ISO: Overlap of Computation and Communication within Seqenence For LLM Inference](https://mlsanity.com/paper/2409.11155/) (2024)
- [Kareus: Joint Reduction of Dynamic and Static Energy in Large Model Training](https://mlsanity.com/paper/2601.17654/) (2026)
- [ForestColl: Throughput-Optimal Collective Communications on Heterogeneous Network Fabrics](https://mlsanity.com/paper/2402.06787/) (2024)
- [Efficient AllReduce with Stragglers](https://mlsanity.com/paper/2505.23523/) (2025)
- [Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts](https://mlsanity.com/paper/2607.19539/) (2026)
- [Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMM](https://mlsanity.com/paper/2410.07531/) (2024)
- [Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs](https://mlsanity.com/paper/2605.05607/) (2026)
- [MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems](https://mlsanity.com/paper/2605.05888/) (2026)
- [Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads](https://mlsanity.com/paper/2606.09200/) (2026)
- [mKernel: Fast Multi-GPU, Multi-Node Fused Kernels](https://mlsanity.com/paper/2609.13585/) (2026)
- [Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs](https://mlsanity.com/paper/2609.40093/) (2026)
- [T-CCL: Resource Efficient and Performant Collective Communication using Tensor Memory Accelerator](https://mlsanity.com/paper/2610.07098/) (2026)

## Abstract (arXiv)

Large Language Models increasingly rely on distributed techniques for their training and inference. These techniques require communication across devices which can reduce scaling efficiency as the number of devices increases. While some distributed techniques can overlap, and thus, hide this communication with independent computations, techniques such as Tensor Parallelism (TP) inherently serialize communication with model execution. One approach to hide this serialized communication is to interleave it with the producer operation (of the communicated data) in a fine-grained manner. However, this fine-grained interleaving of communication and computation in software can be difficult. Furthermore, as with any concurrent execution, it requires compute and memory resources to be shared between computation and communication, causing resource contention that reduces overlapping efficacy. To overcome these challenges, we propose T3 which applies hardware-software co-design to transparently overlap serialized communication while minimizing resource contention with compute. T3 transparently fuses producer operations with the subsequent communication via a simple configuration of the producer's output address space and requires minor software changes. At the hardware level, T3 adds a lightweight track and trigger mechanism to orchestrate the producer's compute, and communication. It further uses compute-enhanced memories for communication's attendant compute. As a result, T3 reduces resource contention, and efficiently overlaps serialized communication with computation. For important Transformer models like T-NLG, T3 speeds up communication-heavy sublayers by 30% geomean (max 47%) and reduces data movement by 22% geomean (max 36%). Furthermore, T3's benefits persist as models scale: geomean 29% for sublayers in ~500-billion parameter models, PALM and MT-NLG.