mlsanityData to 7 Oct 2026

Character-Level Language Modeling with Deeper Self-Attention

arXiv 1808.04444 ↗First posted 9 Aug 2018cs.CL cs.AI cs.LGcited by 138

Not on the map. Why?

Abstract (arXiv)

LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 30; 2023: 24; 2024: 26; 2025: 16; 2026: 6.

138 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  3. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  4. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  5. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  6. 2021A Survey of Transformerscited by 399
  7. 2021Choose a Transformer: Fourier or Galerkincited by 333
  8. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  9. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 229
  10. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  11. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  12. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
38 more
  1. 2023End-to-End Speech Recognition: A Surveycited by 127
  2. 2022Training Language Models with Memory Augmentationcited by 112
  3. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  4. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
  5. 2021Efficient Nearest Neighbor Language Modelscited by 93
  6. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  7. 2021OperA: Attention-Regularized Transformers for Surgical Phase Recognitioncited by 60
  8. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  9. 2023nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Modelscited by 57
  10. 2022Interaction Transformer for Human Reaction Generationcited by 42
  11. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 35
  12. 2023Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionalscited by 31
  13. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  14. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  15. 2021On the limit of English conversational speech recognitioncited by 30
  16. 2021You should evaluate your language model on marginal likelihood over tokenisationscited by 29
  17. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  18. 2021Improving Transformers with Probabilistic Attention Keyscited by 28
  19. 2021Dynamic Language Models for Continuously Evolving Contentcited by 27
  20. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  21. 2021FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attentioncited by 26
  22. 2022TOIST: Task Oriented Instance Segmentation Transformer with Noun-Pronoun Distillationcited by 25
  23. 2023LoCoNet: Long-Short Context Network for Active Speaker Detectioncited by 24
  24. 2025MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologiescited by 23
  25. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  26. 2023Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimationcited by 21
  27. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
  28. 2022PACE: A Parallelizable Computation Encoder for Directed Acyclic Graphscited by 20
  29. 2022Parallel Instance Query Network for Named Entity Recognitioncited by 19
  30. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  31. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
  32. 2021Training Multilingual Pre-trained Language Model with Byte-level Subwordscited by 16
  33. 2021The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architecturescited by 16
  34. 2024Training LLMs over Neurally Compressed Textcited by 16
  35. 2021Evaluating Various Tokenizers for Arabic Text Classificationcited by 14
  36. 2021Staircase Attention for Recurrent Processing of Sequencescited by 13
  37. 2021Security Vulnerability Detection Using Deep Learning Natural Language Processingcited by 12
  38. 2023Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Modelscited by 12

50 most cited of 138 All 138 (JSON)