mlsanityData to 7 Oct 2026

Adaptive Input Representations for Neural Language Modeling

arXiv 1809.10853 ↗First posted 28 Sep 2018cs.CLcited by 226

Not on the map. Why?

Abstract (arXiv)

We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WikiText-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the Billion Word benchmark, we achieve 23.02 perplexity.

Key facts. Cited by 226 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 54; 2022: 49; 2023: 42; 2024: 46; 2025: 21; 2026: 14.

226 papers.

Sort:

By area:

  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  6. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  7. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  8. 2021A Survey of Transformerscited by 399
  9. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  10. 2021Random Feature Attentioncited by 350
  11. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  12. 20213D Human Pose Estimation with Spatial and Temporal Transformerscited by 271
38 more
  1. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  2. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  3. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  4. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  5. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  6. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  7. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 168
  8. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  9. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  10. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  11. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  12. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  13. 2021Unsupervised Speech Recognitioncited by 147
  14. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  15. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  16. 2022Training Language Models with Memory Augmentationcited by 112
  17. 2021Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphscited by 109
  18. 2021Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsitycited by 98
  19. 2021Finetuning Pretrained Transformers into RNNscited by 94
  20. 2021Efficient Nearest Neighbor Language Modelscited by 93
  21. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  22. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  23. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  24. 2021Adaptive Semiparametric Language Modelscited by 83
  25. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  26. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  27. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  28. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  29. 2021Language Model Evaluation Beyond Perplexitycited by 77
  30. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  31. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 71
  32. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  33. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  34. 2022Video Transformers: A Surveycited by 63
  35. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  36. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  37. 2025Large language models for automated scholarly paper review: A surveycited by 59
  38. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Modelcited by 58

50 most cited of 226 All 226 (JSON)