mlsanityData to 7 Oct 2026

Dispatcher: A Message-Passing Approach To Language Modelling

arXiv 2105.03994 ↗First posted 9 May 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper proposes a message-passing mechanism to address language modelling. A new layer type is introduced that aims to substitute self-attention for unidirectional sequence generation tasks. The system is shown to be competitive with existing methods: Given N tokens, the computational complexity is O(N logN) and the memory complexity is O(N) under reasonable assumptions. In the end, the Dispatcher layer is seen to achieve comparable perplexity to prior results while being more efficient.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 9 May 2021.

Builds on

25 of 37 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
13 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  4. 2017Convolutional Sequence to Sequence Learningcited by 550
  5. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  6. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  7. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  8. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  9. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  10. 2019Language Models with Transformerscited by 36
  11. 2020ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesiscited by 34
  12. 2020Let's Agree to Degree: Comparing Graph Convolutional Networks in the Message-Passing Frameworkcited by 24
  13. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23

Built on by

Not cited yet.