mlsanityData to 14 Oct 2026

TorchScale: Transformers at Scale

arXiv 2211.13184 ↗First posted 23 Nov 2022cs.LG cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Large Transformers have achieved state-of-the-art performance across many tasks. Most open-source libraries on scaling Transformers focus on improving training or inference with better parallelization. In this work, we present TorchScale, an open-source toolkit that allows researchers and developers to scale up Transformers efficiently and effectively. TorchScale has the implementation of several modeling techniques, which can improve modeling generality and capability, as well as training stability and efficiency. Experimental results on language modeling and neural machine translation demonstrate that TorchScale can successfully scale Transformers to different sizes without tears. The library is available at https://aka.ms/torchscale.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 12 of its 16 references that are in the map. First posted on arXiv 23 Nov 2022.

Builds on

12 of 16 references.

References · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  8. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  9. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  10. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  11. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  12. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 6; 2024: 2; 2025: 1.

10 papers.

  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  2. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  3. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  4. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  5. 2022A Length-Extrapolatable Transformercited by 127
  6. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  7. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 50
  8. 2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 48
  9. 2024AlignRec: Aligning and Training in Multimodal Recommendationscited by 21
  10. 2025RUMAA: Repeat-Aware Unified Music Audio Analysis for Score-Performance Alignment, Transcription, and Mistake Detectioncited by 0