mlsanityData to 7 Oct 2026

N-Grammer: Augmenting Transformers with latent n-grams

arXiv 2207.06366 ↗First posted 13 Jul 2022cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Transformer models have recently emerged as one of the foundational models in natural language processing, and as a byproduct, there is significant recent interest and investment in scaling these models. However, the training and inference costs of these large Transformer language models are prohibitive, thus necessitating more research in identifying more efficient variants. In this work, we propose a simple yet effective modification to the Transformer architecture inspired by the literature in statistical language modeling, by augmenting the model with n-grams that are constructed from a discrete latent representation of the text sequence. We evaluate our model, the N-Grammer on language modeling on the C4 data-set as well as text classification on the SuperGLUE data-set, and find that it outperforms several strong baselines such as the Transformer and the Primer. We open-source our model for reproducibility purposes in Jax.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 13 Jul 2022.

Builds on

37 of 59 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2016Layer Normalizationcited by 5,579
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  12. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
25 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  9. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  11. 2016Language Modeling with Gated Convolutional Networkscited by 688
  12. 2014Neural Turing Machinescited by 630
  13. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  14. 2014Memory Networkscited by 354
  15. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  16. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  17. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  18. 2019Adaptively Sparse Transformerscited by 154
  19. 2019Large Memory Layers with Product Keyscited by 120
  20. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 103
  21. 2018Theory and Experiments on Vector Quantized Autoencoderscited by 55
  22. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 39
  23. 2021Revisiting Simple Neural Probabilistic Language Modelscited by 10
  24. 2021Lookup-Table Recurrent Language Models for Long Tail Speech Recognitioncited by 9
  25. 2015High Speed Hashing for Integers and Stringscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 2; 2025: 3; 2026: 6.

13 papers.

Sort:

By area:

  1. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  2. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  3. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  4. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  5. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 20
  6. 2025Scaling Embedding Layers in Language Modelscited by 19
  7. 2025Fast and Simplex: 2-Simplicial Attention in Tritoncited by 8
  8. 2024N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needscited by 6
  9. 2024Revisiting N-Gram Models: Their Impact in Modern Neural Networks for Handwritten Text Recognitioncited by 2
  10. 2026Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulationcited by 2
  11. 2026N-gram Injection into Transformers for Dynamic Language Model Adaptation in Handwritten Text Recognitioncited by 0
  12. 2026MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookupcited by 0
1 more
  1. 2026Forking: Sudden Overfitting Under Replaycited by 0