mlsanityData to 7 Oct 2026

Multi-Scale Self-Attention for Text Classification

arXiv 1912.00544 ↗First posted 2 Dec 2019cs.CL cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce the prior knowledge, multi-scale structure, into self-attention modules. We propose a Multi-Scale Transformer which uses multi-scale multi-head self-attention to capture features from different scales. Based on the linguistic perspective and the analysis of pre-trained Transformer (BERT) on a huge corpus, we further design a strategy to control the scale distribution for each layer. Results of three different kinds of tasks (21 datasets) show our Multi-Scale Transformer outperforms the standard Transformer consistently and significantly on small and moderate size datasets.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 5; 2023: 2; 2024: 1; 2025: 1; 2026: 1.

15 papers.

Sort:

  1. 2021A Survey of Transformerscited by 399
  2. 2021Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphscited by 44
  3. 2022Quantum Self-Attention Neural Networks for Text Classificationcited by 42
  4. 2021Code Structure Guided Transformer for Source Code Summarizationcited by 29
  5. 2022M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarizationcited by 14
  6. 2021Cross-lingual Text Classification with Heterogeneous Graph Neural Networkcited by 12
  7. 2023Improving Long Context Document-Level Machine Translationcited by 8
  8. 2022Learning Multiscale Transformer Models for Sequence Generationcited by 7
  9. 2023TranSFormer: Slow-Fast Transformer for Machine Translationcited by 4
  10. 2024CTA-Net: A CNN-Transformer Aggregation Network for Improving Multi-Scale Feature Extractioncited by 4
  11. 2021MUSE: Multi-Scale Temporal Features Evolution for Knowledge Tracingcited by 3
  12. 2022DAGAM: Data Augmentation with Generation And Modificationcited by 2
3 more
  1. 2025MSVIT: Improving Spiking Vision Transformer Using Multi-scale Attention Fusioncited by 2
  2. 2022EIT: Enhanced Interactive Transformercited by 1
  3. 2026Transformer-Enhanced Reinforcement Learning: Fundamentals and Applications in Communication Networkscited by 0