# Multi-Scale Self-Attention for Text Classification

arXiv: https://arxiv.org/abs/1912.00544 · HTML: https://mlsanity.com/paper/1912.00544/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2019.

## Built on by (most cited first, 16 of 16)

- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs](https://mlsanity.com/paper/2110.01191/) (2021)
- [Quantum Self-Attention Neural Networks for Text Classification](https://mlsanity.com/paper/2205.05625/) (2022)
- [Code Structure Guided Transformer for Source Code Summarization](https://mlsanity.com/paper/2104.09340/) (2021)
- [M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarization](https://mlsanity.com/paper/2203.09707/) (2022)
- [Cross-lingual Text Classification with Heterogeneous Graph Neural Network](https://mlsanity.com/paper/2105.11246/) (2021)
- [Improving Long Context Document-Level Machine Translation](https://mlsanity.com/paper/2306.05183/) (2023)
- [Learning Multiscale Transformer Models for Sequence Generation](https://mlsanity.com/paper/2206.09337/) (2022)
- [TranSFormer: Slow-Fast Transformer for Machine Translation](https://mlsanity.com/paper/2305.16982/) (2023)
- [CTA-Net: A CNN-Transformer Aggregation Network for Improving Multi-Scale Feature Extraction](https://mlsanity.com/paper/2410.11428/) (2024)
- [MUSE: Multi-Scale Temporal Features Evolution for Knowledge Tracing](https://mlsanity.com/paper/2102.00228/) (2021)
- [DAGAM: Data Augmentation with Generation And Modification](https://mlsanity.com/paper/2204.02633/) (2022)
- [MSVIT: Improving Spiking Vision Transformer Using Multi-scale Attention Fusion](https://mlsanity.com/paper/2505.14719/) (2025)
- [EIT: Enhanced Interactive Transformer](https://mlsanity.com/paper/2212.10197/) (2022)
- [Transformer-Enhanced Reinforcement Learning: Fundamentals and Applications in Communication Networks](https://mlsanity.com/paper/2606.05208/) (2026)
- [VLCQ: Post-training quantization for deep neural networks using variable length coding](https://mlsanity.com/paper/2610.15092/) (2026)

## Abstract (arXiv)

In this paper, we introduce the prior knowledge, multi-scale structure, into self-attention modules. We propose a Multi-Scale Transformer which uses multi-scale multi-head self-attention to capture features from different scales. Based on the linguistic perspective and the analysis of pre-trained Transformer (BERT) on a huge corpus, we further design a strategy to control the scale distribution for each layer. Results of three different kinds of tasks (21 datasets) show our Multi-Scale Transformer outperforms the standard Transformer consistently and significantly on small and moderate size datasets.