mlsanityData to 7 Oct 2026

ENTP: Encoder-only Next Token Prediction

arXiv 2410.01600 ↗First posted 2 Oct 2024cs.LG cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Next-token prediction is conventionally done using decoder-only Transformers with causal attention, as this approach allows for efficient reuse of keys and values. What if we were not compute-limited, should we still use decoder-only Transformers? In this work, we introduce Encoder-only Next Token Prediction (ENTP). We explore the differences between ENTP and decoder-only Transformers in expressive power and complexity, highlighting potential advantages of ENTP in settings with unbounded compute. We introduce the operatornameCount3 task and show, both theoretically and experimentally, that while ENTP can perform this task easily, a decoder-only Transformer cannot. Finally, we empirically demonstrate the superior performance of ENTP across representative tasks where next-token prediction based Transformers can be evaluated, including addition, in-context learning, and language modeling.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 28 of its 32 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

28 of 32 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  12. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
16 more
  1. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  2. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  3. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  4. 2023Looped Transformers as Programmable Computerscited by 224
  5. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  6. 2021Thinking Like Transformerscited by 182
  7. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  8. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  9. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  10. 2023Representational Strengths and Limitations of Transformerscited by 132
  11. 2023Teaching Arithmetic to Small Transformerscited by 121
  12. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  13. 2022A Logic for Expressing Log-Precision Transformerscited by 67
  14. 2022Bidirectional Language Models Are Also Few-shot Learnerscited by 48
  15. 2024One-layer transformers fail to solve the induction heads taskcited by 30
  16. 2023CausalLM is not optimal for in-context learningcited by 26

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024Theoretical limitations of multi-layer Transformercited by 43
  2. 2024GPT or BERT: why not both?cited by 26
  3. 2025Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecturecited by 16
  4. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  5. 2025Causal Attention with Lookahead Keyscited by 0