mlsanityData to 7 Oct 2026

Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers

arXiv 2402.12233 ↗First posted 19 Feb 2024cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

The feed-forward networks (FFNs) in transformers are recognized as a group of key-value neural memories to restore abstract high-level knowledge. In this work, we conduct an empirical ablation study on updating keys (the 1st layer in the FFNs layer) or values (the 2nd layer in the FFNs layer). We compare those two methods in various knowledge editing and fine-tuning tasks of large language models to draw insights to understand FFNs further. Code is available at hrefhttps://github.com/qiuzh20/Tuning-keys-v.s.-valuesthis,repo.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 11 of its 14 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

11 of 14 references.

References · 11

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2022Mass-Editing Memory in a Transformercited by 976
  7. 2021Editing Factual Knowledge in Language Modelscited by 548
  8. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  9. 2023Transformer-Patcher: One Mistake worth One Neuroncited by 196
  10. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  11. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 1.

8 papers.

  1. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  2. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  3. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  4. 2024GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theorycited by 8
  5. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
  6. 2024Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspectivecited by 4
  7. 2025PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attentioncited by 4
  8. 2025ETT: Expanding the Long Context Understanding Capability of LLMs at Test-Timecited by 1