mlsanityData to 14 Oct 2026

Discovering Latent Knowledge in Language Models Without Supervision

arXiv 2212.03827 ↗First posted 7 Dec 2022cs.CL cs.AI cs.LGcited by 767

Not on the map. Why?

Cited by 100 in Mechanistic interpretability, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.

Key facts. Cited by 767 papers in the map (data through 14 Oct 2026). Builds on 47 of its 74 references that are in the map. First posted on arXiv 7 Dec 2022.

Builds on

47 of 74 references.

References · 47

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
35 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2017Decoupled Weight Decay Regularizationcited by 1,831
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  13. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  14. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  15. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  16. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  17. 2022Red Teaming Language Models with Language Modelscited by 1,026
  18. 2021Ethical and social risks of harm from Language Modelscited by 966
  19. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  20. 2020Recipes for building an open-domain chatbotcited by 593
  21. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  22. 2021MetaICL: Learning to Learn In Contextcited by 454
  23. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  24. 2018AI safety via debatecited by 400
  25. 2021Unsolved Problems in ML Safetycited by 292
  26. 2022Teaching language models to support answers with verified quotescited by 292
  27. 2021Alignment of Language Agentscited by 168
  28. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  29. 2018Supervising strong learners by amplifying weak expertscited by 145
  30. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  31. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  32. 2021Truthful AI: Developing and governing AI that does not liecited by 113
  33. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 67
  34. 2020Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Startcited by 44
  35. 2020Human-Level Performance in No-Press Diplomacy via Equilibrium Searchcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 70; 2024: 151; 2025: 191; 2026: 355.

767 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  4. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  5. 2023Large Language Modelscited by 780
  6. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  7. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  9. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  11. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  12. 2023Language Models Represent Space and Timecited by 316
38 more
  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  2. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  3. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  4. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  5. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  6. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  7. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  8. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  9. 2023An Overview of Catastrophic AI Riskscited by 222
  10. 2024Chain-of-Thought Reasoning Without Promptingcited by 210
  11. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
  12. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  13. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  14. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  15. 2023Model evaluation for extreme riskscited by 177
  16. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
  17. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  18. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  19. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 159
  20. 2023Getting aligned on representational alignmentcited by 143
  21. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 143
  22. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 142
  23. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 141
  24. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  25. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  26. 2024To Believe or Not to Believe Your LLMcited by 120
  27. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  28. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 114
  29. 2025Confidence Improves Self-Consistency in LLMscited by 113
  30. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 105
  31. 2024Universal Neurons in GPT2 Language Modelscited by 102
  32. 2023Deception Abilities Emerged in Large Language Modelscited by 101
  33. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 100
  34. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 99
  35. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 98
  36. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 91
  37. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
  38. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87

50 most cited of 767 All 767 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Calibrating Reasoning in Language Models with Internal Consistency
  2. 2026Uncovering Latent Reasoning Strategies in Language Models
  3. 2025Latent Principle Discovery for Language Model Self-Improvement
  4. 2025InnerThoughts: Disentangling Representations and Predictions in Large Language Models
  5. 2024Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations
  6. 2025ActivationReasoning: Logical Reasoning in Latent Activation Spaces
  7. 2024Strong hallucinations from negation and how to fix them
  8. 2026MechELK: A Mechanistic Interpretability Framework for Eliciting Latent Knowledge in Large Language Models