mlsanityData to 7 Oct 2026

Inspecting and Editing Knowledge Representations in Language Models

arXiv 2304.00740 ↗First posted 3 Apr 2023cs.CLcited by 140

Not on the map. Why?

Abstract (arXiv)

Neural language models (LMs) represent facts about the world described by text. Sometimes these facts derive from training data (in most LMs, a representation of the word "banana" encodes the fact that bananas are fruits). Sometimes facts derive from input text itself (a representation of the sentence "I poured out the bottle" encodes the fact that the bottle became empty). We describe REMEDI, a method for learning to map statements in natural language to fact encodings in an LM's internal representation system. REMEDI encodings can be used as knowledge editors: when added to LM hidden representations, they modify downstream generation to be consistent with new facts. REMEDI encodings may also be used as probes: when compared to LM representations, they reveal which properties LMs already attribute to mentioned entities, in some cases making it possible to predict when LMs will generate outputs that conflict with background knowledge or input text. REMEDI thus links work on probing, prompting, and LM editing, and offers steps toward general tools for fine-grained inspection and control of knowledge in LMs.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

39 of 55 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2022Mass-Editing Memory in a Transformercited by 976
27 more
  1. 2019How Can We Know What Language Models Know?cited by 953
  2. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  4. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  5. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  6. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  7. 2018CoQA: A Conversational Question Answering Challengecited by 674
  8. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  9. 2019Designing and Interpreting Probes with Control Taskscited by 608
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  11. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  12. 2021Editing Factual Knowledge in Language Modelscited by 548
  13. 2021Fast Model Editing at Scalecited by 491
  14. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  15. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  16. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  17. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  18. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  19. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  20. 2023A Categorical Archive of ChatGPT Failurescited by 167
  21. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 144
  22. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  23. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  24. 2021Tailor: Generating and Perturbing Text with Semantic Controlscited by 67
  25. 2022COPEN: Probing Conceptual Knowledge in Pre-trained Language Modelscited by 21
  26. 2019What Does My QA Model Know? Devising Controlled Probes using Expert Knowledgecited by 19
  27. 2018Semantic projection: recovering human knowledge of multiple, distinct object features from word embeddingscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 49; 2025: 48; 2026: 28.

140 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  3. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  4. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  5. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  6. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  7. 2023PMET: Precise Model Editing in a Transformercited by 170
  8. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  9. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  10. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 97
  11. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
  12. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 79
38 more
  1. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  2. 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 63
  3. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  4. 2024Semantics-Adaptive Activation Intervention for LLMs via Dynamic Steering Vectorscited by 61
  5. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  6. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 49
  7. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 47
  8. 2023Can We Edit Multimodal Large Language Models?cited by 44
  9. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspectivecited by 44
  10. 2024Steering Without Side Effects: Improving Post-Deployment Control of Language Modelscited by 44
  11. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 39
  12. 2025LLM Unlearning via Neural Activation Redirectioncited by 38
  13. 2024The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Unitscited by 34
  14. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  15. 2024Activation Scaling for Steering and Interpreting Language Modelscited by 33
  16. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  17. 2024Backtracking Improves Generation Safetycited by 31
  18. 2024Towards Unifying Interpretability and Control: Evaluation via Interventioncited by 28
  19. 2024PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restorationcited by 27
  20. 2025One-shot Optimized Steering Vectors Mediate Safety-relevant Behaviors in LLMscited by 27
  21. 2024Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methodscited by 26
  22. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  23. 2024Householder Pseudo-Rotation: A Novel Approach to Activation Editing in LLMs with Direction-Magnitude Perspectivecited by 23
  24. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Modelscited by 23
  25. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspectivecited by 22
  26. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  27. 2024SafeInfer: Context Adaptive Decoding Time Safety Alignment for Large Language Modelscited by 22
  28. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  29. 2024A Survey on the Honesty of Large Language Modelscited by 21
  30. 2024LiSeCo: Linear Semantic Control for Language Generationcited by 20
  31. 2023DUnE: Dataset for Unified Editingcited by 19
  32. 2024A Causal Explainable Guardrails for Large Language Modelscited by 18
  33. 2024CodeUpdateArena: Benchmarking Knowledge Editing on API Updatescited by 18
  34. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  35. 2025Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation Steeringcited by 18
  36. 2025Emergent Response Planning in LLMscited by 17
  37. 2025Unlocking Efficient, Scalable, and Continual Knowledge Editing with Basis-Level Representation Fine-Tuningcited by 17
  38. 2026Spherical Steering: Geometry-Aware Activation Rotation for Language Modelscited by 17

50 most cited of 140 All 140 (JSON)