mlsanityData to 7 Oct 2026

Star-Transformer

arXiv 1902.09113 ↗First posted 25 Feb 2019cs.CLcited by 94

Not on the map. Why?

Abstract (arXiv)

Although Transformer has achieved great successes on many NLP tasks, its heavy structure with fully-connected attention connections leads to dependencies on large training data. In this paper, we present Star-Transformer, a lightweight alternative by careful sparsification. To reduce model complexity, we replace the fully-connected structure with a star-shaped topology, in which every two non-adjacent nodes are connected through a shared relay node. Thus, complexity is reduced from quadratic to linear, while preserving capacity to capture both local composition and long-range dependency. The experiments on four tasks (22 datasets) show that Star-Transformer achieved significant improvements against the standard Transformer for the modestly sized datasets.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 18; 2023: 16; 2024: 18; 2025: 12; 2026: 2.

94 papers.

Sort:

By area:

  1. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  2. 2021A Survey of Transformerscited by 399
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  5. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  6. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  7. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  8. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  9. 2021Vision Transformer Pruningcited by 77
  10. 2024SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusioncited by 75
  11. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  12. 2022NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesiscited by 71
38 more
  1. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  2. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  3. 2021Syntax-BERT: Improving Pre-trained Transformers with Syntax Treescited by 51
  4. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  5. 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
  6. 2021Long-Range Transformers for Dynamic Spatiotemporal Forecastingcited by 46
  7. 2024A Survey on Transformer Compressioncited by 44
  8. 2022Relational Attention: Generalizing Transformers for Graph-Structured Taskscited by 41
  9. 2021A Unified Pruning Framework for Vision Transformerscited by 40
  10. 2021Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modelingcited by 35
  11. 2022MTTrans: Cross-Domain Object Detection with Mean-Teacher Transformercited by 35
  12. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  13. 2021Code Structure Guided Transformer for Source Code Summarizationcited by 29
  14. 2022Transformers from an Optimization Perspectivecited by 27
  15. 2023Escaping the sentence-level paradigm in machine translationcited by 27
  16. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMscited by 27
  17. 2021How to Leverage Multimodal EHR Data for Better Medical Predictions?cited by 26
  18. 2022Multimodal Pre-training Based on Graph Attention Network for Document Understandingcited by 23
  19. 2022Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networkscited by 23
  20. 2024CaDA: Cross-Problem Routing Solver with Constraint-Aware Dual-Attentioncited by 23
  21. 2024TransformerFAM: Feedback attention is working memorycited by 21
  22. 2021MATCH: Metadata-Aware Text Classification in A Large Hierarchycited by 20
  23. 2023AI Foundation Models for Weather and Climate: Applications, Design, and Implementationcited by 19
  24. 2022Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioningcited by 18
  25. 2022PuzzleFusion: Unleashing the Power of Diffusion Models for Spatial Puzzle Solvingcited by 17
  26. 2022LSG Attention: Extrapolation of pretrained Transformers to long sequencescited by 16
  27. 2023FIT: Far-reaching Interleaved Transformerscited by 16
  28. 2024TrafficGPT: Breaking the Token Barrier for Efficient Long Traffic Analysis and Generationcited by 15
  29. 2023Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformerscited by 13
  30. 2024MSHyper: Multi-Scale Hypergraph Transformer for Long-Range Time Series Forecastingcited by 12
  31. 2025RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routingcited by 11
  32. 2022SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequencescited by 10
  33. 2022CAB: Comprehensive Attention Benchmarking on Long Sequence Modelingcited by 10
  34. 2021MotifClass: Weakly Supervised Text Classification with Higher-order Metadata Informationcited by 9
  35. 2022Unsupervised Spike Depth Estimation via Cross-modality Cross-domain Knowledge Transfercited by 8
  36. 2021TENT: Tensorized Encoder Transformer for Temperature Forecastingcited by 7
  37. 2025Prior-Fitted Networks Scale to Larger Datasets When Treated as Weak Learnerscited by 7
  38. 2021Eigen Analysis of Self-Attention and its Reconstruction from Partial Computationcited by 6

50 most cited of 94 All 94 (JSON)