mlsanityData to 14 Oct 2026

White-Box Sensitivity Auditing with Steering Vectors

arXiv 2601.16398 ↗First posted 23 Jan 2026cs.CY cs.CL cs.LGcited by 0

On the map: Mechanistic interpretability → Probing deception, bias and beliefs

Abstract (arXiv)

Algorithmic audits are essential tools for examining systems for properties required by regulators or desired by operators. Current audits of large language models (LLMs) primarily rely on black-box evaluations that assess model behavior only through input-output testing. These methods are limited to tests constructed in the input space, often generated by heuristics. In addition, many socially relevant model properties (e.g., gender bias) are abstract and difficult to measure through text-based inputs alone. To address these limitations, we propose a white-box sensitivity auditing framework for LLMs that leverages activation steering to conduct more rigorous assessments through model internals. Our auditing method conducts internal sensitivity tests by manipulating key concepts relevant to the model's intended function for the task. We demonstrate its application to bias audits in four simulated high-stakes LLM decision tasks. Our method consistently indicates substantial dependence on protected attributes in model predictions, even in settings where standard black-box evaluations suggest little or no bias. Our code is openly available at https://github.com/hannahxchen/llm-steering-audit

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Probing deception, bias and beliefs, one of 13 directions in Mechanistic interpretability. Builds on 56 of its 78 references that are in the map. First posted on arXiv 23 Jan 2026.

Builds on

56 of 78 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective5 Jun
  2. 2025Unsupervised Concept Vector Extraction for Bias Control in LLMs27 Feb

Other references · 54

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  6. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  7. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  8. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  9. 2023Steering Language Models With Activation Engineeringcited by 922
  10. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  11. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  12. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
38 more
  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  2. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  3. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  5. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  6. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  7. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  8. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  9. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  10. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
  11. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  12. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  13. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  14. 2019Certified Robustness to Adversarial Word Substitutionscited by 168
  15. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 151
  16. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
  17. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
  18. 2023Auditing large language models: a three-layered approachcited by 137
  19. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  20. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 124
  21. 2022The Fallacy of AI Functionalitycited by 88
  22. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  23. 2024On the Origins of Linear Representations in Large Language Modelscited by 79
  24. 2019Perturbation Sensitivity Analysis to Detect Unintended Model Biasescited by 74
  25. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  26. 2020Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbationscited by 55
  27. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 53
  28. 2024What's in a Name? Auditing Large Language Models for Race and Gender Biascited by 51
  29. 2024Social Bias Evaluation for Large Language Models Requires Prompt Variationscited by 40
  30. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  31. 2020The Problem with Metrics is a Fundamental Problem for AIcited by 37
  32. 2018Adversarial Over-Sensitivity and Over-Stability Strategies for Dialogue Modelscited by 26
  33. 2025We Can't Understand AI Using our Existing Vocabularycited by 25
  34. 2025Privacy Auditing of Large Language Modelscited by 23
  35. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  36. 2025Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Controlcited by 14
  37. 2022An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiringcited by 10
  38. 2023Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domainscited by 8

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Activation Steering for Bias Mitigation: An Interpretable Approach to Safer LLMs
  2. 2025Shifting Perspectives: Steering Vectors for Robust Bias Mitigation in LLMs
  3. 2024An Auditing Test To Detect Behavioral Shift in Language Models
  4. 2025Auditing language models for hidden objectives
  5. 2025Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations
  6. 2025Beyond Black-Box Obfuscation: Mechanistic Analysis and Defense of White-Box Monitors
  7. 2026Latent space bias directions in LLMs capture confidence, not fairness
  8. 2026Steered LLM Activations are Non-Surjective