mlsanityData to 7 Oct 2026

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

arXiv 1901.02860 ↗First posted 9 Jan 2019cs.LG cs.CL stat.MLcited by 1,830

Not on the map. Why?

Cited by 43 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.

Key facts. Cited by 1,830 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 2; 2021: 361; 2022: 325; 2023: 353; 2024: 326; 2025: 239; 2026: 223.

1,830 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  10. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
38 more
  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  6. 2022A Generalist Agentcited by 802
  7. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  9. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  11. 2021GPT Understands, Toocited by 715
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  13. 2022Simple Baselines for Image Restorationcited by 701
  14. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  15. 2023A Comprehensive Overview of Large Language Modelscited by 639
  16. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  17. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  18. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  19. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  20. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  21. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  22. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  23. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  24. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  25. 2022Transformers in Time Series: A Surveycited by 489
  26. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  27. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  28. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  29. 2021A Survey of Transformerscited by 399
  30. 2022Multimodal Learning with Transformers: A Surveycited by 399
  31. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  32. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  33. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  34. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  35. 2021Random Feature Attentioncited by 350
  36. 2023Adapting Language Models to Compress Contextscited by 350
  37. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  38. 2021Efficient Attentions for Long Document Summarizationcited by 342

50 most cited of 1,830 All 1,830 (JSON)