mlsanityData to 7 Oct 2026

SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models

arXiv 2502.20727 ↗First posted 28 Feb 2025cs.DC cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

With the rapid expansion in the scale of large language models (LLMs), enabling efficient distributed inference across multiple computing units has become increasingly critical. However, communication overheads from popular distributed inference techniques such as Tensor Parallelism pose a significant challenge to achieve scalability and low latency. Therefore, we introduce a novel optimization technique, Sync-Point Drop (SPD), to reduce communication overheads in tensor parallelism by selectively dropping synchronization on attention outputs. In detail, we first propose a block design that allows execution to proceed without communication through SPD. Second, we apply different SPD strategies to attention blocks based on their sensitivity to the model accuracy. The proposed methods effectively alleviate communication bottlenecks while minimizing accuracy degradation during LLM inference, offering a scalable solution for diverse distributed environments: SPD offered about 20% overall inference latency reduction with < 1% accuracy regression for LLaMA2-70B inference over 8 GPUs.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 33 of its 37 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

33 of 37 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
21 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  8. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  11. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  13. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  14. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  15. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  16. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 96
  17. 2024CHAI: Clustered Head Attention for Efficient LLM Inferencecited by 13
  18. 2024eXmY: A Data Type and Technique for Arbitrary Bit Precision Quantizationcited by 7
  19. 2024KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generationcited by 6
  20. 2024Towards Low-bit Communication for Tensor Parallel LLM Inferencecited by 6
  21. 2023ATP: Adaptive Tensor Parallelism for Foundation Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Tensor-Parallelism with Partially Synchronized Activationscited by 4
  2. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  3. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inferencecited by 1
  4. 2025A Unified Sparse Attention via Multi-Granularity Compressioncited by 1
  5. 2026Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronizationcited by 1
  6. 2026Affinity-Aware Sharding for Delayed Tensor Parallelismcited by 0