mlsanityData to 7 Oct 2026

Using the Output Embedding to Improve Language Models

arXiv 1608.05859 ↗First posted 20 Aug 2016cs.CLcited by 266

Not on the map. Why?

Abstract (arXiv)

We study the topmost weight matrix of neural network language models. We show that this matrix constitutes a valid word embedding. When training language models, we recommend tying the input embedding and this output embedding. We analyze the resulting update rules and show that the tied embedding evolves in a more similar way to the output embedding than to the input embedding in the untied model. We also offer a new method of regularizing the output embedding. Our methods lead to a significant reduction in perplexity, as we are able to show on a variety of neural network language models. Finally, we show that weight tying can reduce the size of neural translation models to less than half of their original size without harming their performance.

Key facts. Cited by 266 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Aug 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 47; 2022: 41; 2023: 42; 2024: 44; 2025: 34; 2026: 57.

266 papers.

Sort:

By area:

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  5. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  7. 2023The Falcon Series of Open Language Modelscited by 526
  8. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  9. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  10. 2022Continuous diffusion for categorical datacited by 202
  11. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  12. 2021Training Graph Neural Networks with 1000 Layerscited by 169
38 more
  1. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  2. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  3. 2022Analyzing Transformers in Embedding Spacecited by 131
  4. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  5. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  6. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
  7. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  8. 2021Finetuning Pretrained Transformers into RNNscited by 94
  9. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  10. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  11. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  12. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  13. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  14. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  15. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  16. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  17. 2023Trajeglish: Traffic Modeling as Next-Token Predictioncited by 59
  18. 2021VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learningcited by 56
  19. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  20. 2021A Survey on Green Deep Learningcited by 52
  21. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  22. 2025Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokenscited by 48
  23. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  24. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  25. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 40
  26. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 39
  27. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  28. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  29. 2024From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformerscited by 34
  30. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  31. 2021Many-to-English Machine Translation Tools, Data, and Pretrained Modelscited by 31
  32. 2023Gloss-Free End-to-End Sign Language Translationcited by 31
  33. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 31
  34. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
  35. 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
  36. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  37. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  38. 2022Efficient recurrent architectures through activity sparsity and sparse back-propagation through timecited by 26

50 most cited of 266 All 266 (JSON)