mlsanityData to 7 Oct 2026

Attention Is All You Need

arXiv 1706.03762 ↗First posted 12 Jun 2017cs.CL cs.LGcited by 59,932

Not on the map. Why?

Cited by 2,672 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

Key facts. Cited by 59,932 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 4; 2019: 4; 2020: 17; 2021: 6660; 2022: 8621; 2023: 11311; 2024: 12804; 2025: 12156; 2026: 8355.

59,932 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2023Segment Anythingcited by 8,158
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  13. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  14. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  15. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  16. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  17. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  18. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  19. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  20. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  21. 2022A ConvNet for the 2020scited by 3,545
  22. 2021Program Synthesis with Large Language Modelscited by 3,509
  23. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  24. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  25. 2023Mistral 7Bcited by 3,065
  26. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  27. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  28. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  29. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  30. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  31. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  32. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  33. 2020Learning to summarize from human feedbackcited by 2,562
  34. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  35. 2023A Survey of Large Language Modelscited by 2,537
  36. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  37. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  38. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388

50 most cited of 59,932 All 59,932 (JSON)