mlsanityData to 14 Oct 2026

ASTRA: Communication-Efficient Acceleration for Multi-Device Transformer Inference

arXiv 2505.19342 ↗First posted 25 May 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-device inference can reduce Transformer latency by parallelizing computation. However, existing methods require high inter-device bandwidth, making them impractical for bandwidth-constrained environments. We present ASTRA, a communication-efficient framework that integrates sequence parallelism with mixed-precision attention, where non-local token embeddings are transmitted as low-bit vector-quantized codes while local attention remains full precision. To preserve accuracy under aggressive compression, ASTRA introduces Noise-Augmented Quantization and Distributed Class Tokens. Across vision and language models (e.g., ViT and GPT2), ASTRA achieves up to 2.64× speedup over single-device inference and up to 15.25× over prior multi-device baselines while operating at bandwidths as low as 10 Mbps. ASTRA remains robust on large models (e.g., Llama-3-8B) even under non-ideal network conditions such as packet loss and dynamic networks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

21 of 40 references.

References · 21

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2019Are Sixteen Heads Really Better than One?cited by 816
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  12. 2019Q8BERT: Quantized 8Bit BERTcited by 300
9 more
  1. 2021Post-Training Quantization for Vision Transformercited by 278
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  3. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  4. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 97
  5. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 72
  6. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  7. 2020CoEdge: Cooperative DNN Inference with Adaptive Workload Partitioning over Heterogeneous Edge Devicescited by 67
  8. 2019Natural Language Understanding with the Quora Question Pairs Datasetcited by 51
  9. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025EdgeFlex-Transformer: Transformer Inference for Edge Devices
  2. 2026Transformer Accelerator (TFA): A Macro-Op INT8 Hardware Chip for Transformer Inference and Machine Translation
  3. 2022Efficiently Scaling Transformer Inference
  4. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding
  5. 2023AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems
  6. 2025End-to-End Transformer Acceleration Through Processing-in-Memory Architectures
  7. 2025LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference
  8. 2025Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edge