mlsanityData to 7 Oct 2026

Learning Multiscale Transformer Models for Sequence Generation

arXiv 2206.09337 ↗First posted 19 Jun 2022cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Multiscale feature hierarchies have been witnessed the success in the computer vision area. This further motivates researchers to design multiscale Transformer for natural language processing, mostly based on the self-attention mechanism. For example, restricting the receptive field across heads or extracting local fine-grained features via convolutions. However, most of existing works directly modeled local features but ignored the word-boundary information. This results in redundant and ambiguous attention distributions, which lacks of interpretability. In this work, we define those scales in different linguistic units, including sub-words, words and phrases. We built a multiscale Transformer model by establishing relationships among scales based on word-boundary information and phrase-level prior knowledge. The proposed Universal MultiScale Transformer, namely textscUmst, was evaluated on two sequence generation tasks. Notably, it yielded consistent performance gains over the strong baseline on several test sets without sacrificing the efficiency.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 52 of its 61 references that are in the map. First posted on arXiv 19 Jun 2022.

Builds on

52 of 61 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  7. 2017Graph Attention Networkscited by 8,854
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2016Layer Normalizationcited by 5,579
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
38 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2015Teaching Machines to Read and Comprehendcited by 1,039
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  10. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  11. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  12. 2021Multiscale Vision Transformerscited by 896
  13. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  14. 2021Transformer in Transformercited by 632
  15. 2018Universal Transformerscited by 613
  16. 2019Deep Equilibrium Modelscited by 603
  17. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  18. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  19. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  20. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  21. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  22. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  23. 2018Scaling Neural Machine Translationcited by 204
  24. 2018Constituency Parsing with a Self-Attentive Encodercited by 196
  25. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  26. 2018Bottom-Up Abstractive Summarizationcited by 177
  27. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  28. 2020Vocabulary Learning via Optimal Transport for Neural Machine Translationcited by 58
  29. 2019Universal Dependency Parsing from Scratchcited by 46
  30. 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
  31. 2019Convolutional Self-Attention Networkscited by 32
  32. 2021Mask Attention Networks: Rethinking and Strengthen Transformercited by 28
  33. 2020Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learningcited by 26
  34. 2022ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generationcited by 26
  35. 2020Learning Light-Weight Translation Models from Deep Transformercited by 22
  36. 2019MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learningcited by 19
  37. 2020DeLighT: Deep and Light-weight Transformercited by 17
  38. 2019Multi-Scale Self-Attention for Text Classificationcited by 15

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 5; 2024: 1.

7 papers.

  1. 2023ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generationcited by 15
  2. 2023Learning Evaluation Models from Large Language Models for Sequence Generationcited by 6
  3. 2023TranSFormer: Slow-Fast Transformer for Machine Translationcited by 4
  4. 2023Enhancing Neural Machine Translation with Semantic Unitscited by 2
  5. 2022EIT: Enhanced Interactive Transformercited by 1
  6. 2023Pluggable Neural Machine Translation Models via Memory-augmented Adapterscited by 1
  7. 2024End-to-end Planner Training for Language Modelingcited by 0