mlsanityData to 7 Oct 2026

Understanding Token Probability Encoding in Output Embeddings

arXiv 2406.01468 ↗First posted 3 Jun 2024cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the output token probability information in the output embedding of language models. We find an approximate common log-linear encoding of output token probabilities within the output embedding vectors and empirically demonstrate that it is accurate and sparse. As a causality examination, we steer the encoding in output embedding to modify the output probability distribution accurately. Moreover, the sparsity we find in output probability encoding suggests that a large number of dimensions in the output embedding do not contribute to causal language modeling. Therefore, we attempt to delete the output-unrelated dimensions and find more than 30% of the dimensions can be deleted without significant movement in output distribution and sequence generation. Additionally, in the pre-training dynamics of language models, we find that the output embeddings capture the corpus token frequency information in early steps, even before an obvious convergence of parameters starts.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

38 of 49 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2016Layer Normalizationcited by 5,579
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
26 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  5. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  6. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  7. 2020What is being transferred in transfer learning?cited by 365
  8. 2023Explainability for Large Language Models: A Surveycited by 331
  9. 2016Using the Output Embedding to Improve Language Modelscited by 266
  10. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  11. 2017Real Time Image Saliency for Black Box Classifierscited by 246
  12. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  13. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  14. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  15. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  16. 2020The Early Phase of Neural Network Trainingcited by 114
  17. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  18. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  19. 2023Mitigating Label Biases for In-context Learningcited by 88
  20. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  21. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  22. 2018FRAGE: Frequency-Agnostic Word Representationcited by 47
  23. 2021Interpreting Deep Learning Models in Natural Language Processing: A Reviewcited by 19
  24. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  25. 2023Transformer Language Models Handle Word Frequency in Prediction Headcited by 8
  26. 2022Investigating the Frequency Distortion of Word Embeddings and Its Impact on Bias Metricscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 4.

6 papers.

  1. 2024Probabilistic Medical Predictions of Large Language Modelscited by 16
  2. 2025Measuring Intrinsic Dimension of Token Embeddingscited by 5
  3. 2026Why Does Post-Training Quantization Work?cited by 1
  4. 2026On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decodecited by 0
  5. 2026Reading Without a Reader: Large Language Models Collapse Reading and Writing into a Single Entangled Codecited by 0
  6. 2026The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priorscited by 0