mlsanityData to 7 Oct 2026

Query-Key Normalization for Transformers

arXiv 2010.04245 ↗First posted 8 Oct 2020cs.CL cs.AI cs.LGcited by 266

Not on the map. Why?

Cited by 40 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Low-resource language translation is a challenging but socially valuable NLP task. Building on recent work adapting the Transformer's normalization to this setting, we propose QKNorm, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. Specifically, we apply ℓ2 normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension. We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15.

Key facts. Cited by 266 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 11; 2024: 39; 2025: 78; 2026: 132.

266 papers.

Sort:

By area:

  1. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2024Genie: Generative Interactive Environmentscited by 706
  4. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  5. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  6. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  7. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  8. 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 294
  9. 2022Memorizing Transformerscited by 219
  10. 2026Gemma 4 Technical Reportcited by 187
  11. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
  12. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
38 more
  1. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  2. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  3. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  4. 2025Test-Time Training Done Rightcited by 121
  5. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 116
  6. 2025GR-3 Technical Reportcited by 110
  7. 2022Block-Recurrent Transformerscited by 107
  8. 2026Generative Modeling via Driftingcited by 98
  9. 2025Latent Diffusion Model without Variational Autoencodercited by 83
  10. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 82
  11. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  12. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 74
  13. 2025LongCat-Video Technical Reportcited by 73
  14. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  15. 2024DrivingWorld: Constructing World Model for Autonomous Driving via Video GPTcited by 66
  16. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 65
  17. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  18. 2023TransNeXt: Robust Foveal Visual Perception for Vision Transformerscited by 61
  19. 2025FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policiescited by 56
  20. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  21. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  22. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  23. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Modelscited by 45
  24. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
  25. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 43
  26. 2022How to Understand Masked Autoencoderscited by 37
  27. 2025DeepVerse: 4D Autoregressive Video Generation as a World Modelcited by 36
  28. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 33
  29. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  30. 2024MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Modelcited by 31
  31. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformercited by 30
  32. 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agentscited by 30
  33. 2021Attention Approximates Sparse Distributed Memorycited by 29
  34. 2026Controlled LLM Training on Spectral Spherecited by 28
  35. 2024SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Trainingcited by 27
  36. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  37. 2025Absolute Coordinates Make Motion Generation Easycited by 27
  38. 2026ELF: Embedded Language Flowscited by 27

50 most cited of 266 All 266 (JSON)