mlsanityData to 7 Oct 2026

Eliciting Latent Knowledge from Quirky Language Models

arXiv 2312.01037 ↗First posted 2 Dec 2023cs.LG cs.AI cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

Eliciting Latent Knowledge (ELK) aims to find patterns in a capable neural network's activations that robustly track the true state of the world, especially in hard-to-verify cases where the model's output is untrusted. To further ELK research, we introduce 12 datasets and a corresponding suite of "quirky" language models (LMs) that are finetuned to make systematic errors when answering questions if and only if the keyword "Bob" is present in the prompt. We find that, especially in middle layers, linear probes usually report an LM's knowledge independently of what the LM outputs, enabling us to elicit the correct answer despite the model's untruthful output. The best probing method (logistic regression on contrast pairs) recovers 89% of the gap in AUROC between truthful and untruthful contexts, and 75% for questions harder than those used to train the probe. We also find that a mechanistic anomaly detection approach can flag untruthful behavior with 0.95 AUROC. Our results show promise for eliciting reliable knowledge from capable but untrusted models, and facilitates future research empirically investigating ELK methods.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 24 of its 45 references that are in the map. First posted on arXiv 2 Dec 2023.

Builds on

24 of 45 references.

References · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  12. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
12 more
  1. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  3. 2023Llemma: An Open Language Model For Mathematicscited by 385
  4. 2018AI safety via debatecited by 384
  5. 2022Self-critiquing models for assisting human evaluatorscited by 372
  6. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  7. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  8. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  9. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  10. 2018Supervising strong learners by amplifying weak expertscited by 139
  11. 2018Invariance, Causality and Robustnesscited by 123
  12. 2019exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Modelscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 19; 2026: 19.

56 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  3. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  4. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  5. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  6. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  7. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  8. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
  9. 2024A Language Model's Guide Through Latent Spacecited by 43
  10. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 39
  11. 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localizationcited by 34
  12. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Modelscited by 31
38 more
  1. 2025To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Modelscited by 26
  2. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 24
  3. 2025RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?cited by 19
  4. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  5. 2025Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Taskscited by 18
  6. 2024Token Erasure as a Footprint of Implicit Vocabulary Items in LLMscited by 17
  7. 2024Towards Inference-time Category-wise Safety Steering for Large Language Modelscited by 17
  8. 2025LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probingcited by 14
  9. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 11
  10. 2026Shaping capabilities with token-level data filteringcited by 11
  11. 2024Does Transformer Interpretability Transfer to RNNs?cited by 9
  12. 2025SafeSteer: Interpretable Safety Steering with Refusal-Evasion in LLMscited by 9
  13. 2025JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Modelscited by 8
  14. 2026Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetrycited by 8
  15. 2025Reasoning Under Pressure: How do Training Incentives Influence Chain-of-Thought Monitorability?cited by 7
  16. 2025Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time Alignmentcited by 6
  17. 2026Catching rationalization in the act: detecting motivated reasoning before and after CoT via activation probingcited by 6
  18. 2024Investigating the Indirect Object Identification circuit in Mambacited by 5
  19. 2025CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimizationcited by 5
  20. 2024Cluster-norm for Unsupervised Probing of Knowledgecited by 4
  21. 2025Mechanistic Anomaly Detection for "Quirky" Language Modelscited by 4
  22. 2025Toward a Flexible Framework for Linear Representation Hypothesis Using Maximum Likelihood Estimationcited by 3
  23. 2025Exploring How LLMs Capture and Represent Domain-Specific Knowledgecited by 3
  24. 2026Interpreting and Steering State-Space Models via Activation Subspace Bottleneckscited by 3
  25. 2024Balancing Label Quantity and Quality for Scalable Elicitationcited by 2
  26. 2026Split Personality Training: Revealing Latent Knowledge Through Alternate Personalitiescited by 2
  27. 2026Pando: Do Interpretability Methods Work When Models Won't Explain Themselves?cited by 2
  28. 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actionscited by 2
  29. 2026LLM Layers Immediately Correct Each Othercited by 2
  30. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimizationcited by 1
  31. 2026TrojanPraise: Jailbreak LLMs via Benign Fine-Tuningcited by 1
  32. 2026Mechanistic Anomaly Detection via Functional Attributioncited by 1
  33. 2026The Impossibility of Eliciting Latent Knowledgecited by 1
  34. 2025Towards Integrated Alignmentcited by 0
  35. 2025Who's the Evil Twin? Differential Auditing for Undesired Behaviorcited by 0
  36. 2025RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositoriescited by 0
  37. 2026Intention Collapse: Intention-Level Metrics for Reasoning in Language Modelscited by 0
  38. 2026Three Concrete Challenges and Two Hopes for the Safety of Unsupervised Elicitationcited by 0

50 most cited of 56 All 56 (JSON)