mlsanityData to 7 Oct 2026

Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space

arXiv 2203.14680 ↗First posted 28 Mar 2022cs.CLcited by 522

Not on the map. Why?

Cited by 37 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer-based language models (LMs) are at the core of modern NLP, but their internal prediction construction process is opaque and largely not understood. In this work, we make a substantial step towards unveiling this underlying prediction process, by reverse-engineering the operation of the feed-forward network (FFN) layers, one of the building blocks of transformer models. We view the token representation as a changing distribution over the vocabulary, and the output from each FFN layer as an additive update to that distribution. Then, we analyze the FFN updates in the vocabulary space, showing that each update can be decomposed to sub-updates corresponding to single FFN parameter vectors, each promoting concepts that are often human-interpretable. We then leverage these findings for controlling LM predictions, where we reduce the toxicity of GPT2 by almost 50%, and for improving computation efficiency with a simple early exit rule, saving 20% of computation on average.

Key facts. Cited by 522 papers in the map (data through 7 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 28 Mar 2022.

Builds on

33 of 45 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016Layer Normalizationcited by 5,579
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
21 more
  1. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  2. 2019How Can We Know What Language Models Know?cited by 953
  3. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  4. 2019Green AIcited by 620
  5. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  6. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  7. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  8. 2019Depth-Adaptive Transformercited by 227
  9. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  10. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  11. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  12. 2019Augmenting Self-attention with Persistent Memorycited by 92
  13. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  14. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  15. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  16. 2021A Survey on Green Deep Learningcited by 52
  17. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  18. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  19. 2022Kformer: Knowledge Injection in Transformer Feed-Forward Layerscited by 37
  20. 2021Analyzing Commonsense Emergence in Few-shot Knowledge Modelscited by 34
  21. 2020On the Sub-Layer Functionalities of Transformer Decodercited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 78; 2024: 142; 2025: 135; 2026: 151.

522 papers.

Sort:

By area:

  1. 2022Mass-Editing Memory in a Transformercited by 976
  2. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  3. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  5. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  6. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  7. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  8. 2023Explainability for Large Language Models: A Surveycited by 331
  9. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  10. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  11. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  12. 2022Confident Adaptive Language Modelingcited by 297
38 more
  1. 2023Function Vectors in Large Language Modelscited by 292
  2. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  3. 2023Large Language Model Alignment: A Surveycited by 241
  4. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  5. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  6. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  7. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  8. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  9. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  10. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  11. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  12. 2023PMET: Precise Model Editing in a Transformercited by 170
  13. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  14. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  15. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  16. 2024How do Large Language Models Handle Multilingualism?cited by 148
  17. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  18. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  19. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  20. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  21. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  22. 2022Analyzing Transformers in Embedding Spacecited by 131
  23. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  24. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 116
  25. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  26. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  27. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  28. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescuecited by 101
  29. 2024Universal Neurons in GPT2 Language Modelscited by 100
  30. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  31. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  32. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 97
  33. 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
  34. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  35. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
  36. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  37. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  38. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88

50 most cited of 522 All 522 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
  2. 2026More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
  3. 2026A Single-Layer Model Can Do Language Modeling
  4. 2024Linear Transformers with Learnable Kernel Functions are Better In-Context Models
  5. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers
  6. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models
  7. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  8. 2024A Survey of Transformer-based Language Models with Focus on Efficiency