mlsanityData to 7 Oct 2026

Less Sycophancy, Stronger Refusal? Lessons for AI Safety from Mechanistic Interpretability

arXiv 2609.35544 ↗First posted 28 Sep 2026cs.CL cs.AI cs.LGcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Reliable refusal of harmful requests is essential to the safe deployment of language models. Because excessive eagerness to please users may undermine existing refusal capabilities, reducing sycophancy offers a potential route to stronger refusal beyond the harmful scenarios covered by safety training. We investigate this possibility using compensatory feature injection (CFI), a training technique designed to limit the acquisition of a target concept by supplying its associated activation during learning. Across three Qwen3.5 base models, we use sparse autoencoders (SAEs) to identify the top-ranked sycophancy feature from paired sycophantic and independent responses, then validate its behavioral influence through inference steering. We subsequently inject the selected feature during supervised fine-tuning on sycophantic targets. Positive injection reduces learned sycophancy after removal (by 62.0% relative to ordinary fine-tuning in 35B-A3B), whereas modest negative injection increases it. Unexpectedly, these reductions in sycophancy do not consistently improve direct refusal of harmful requests, motivating a narrower evaluation of the same harmful intents under user pressure. In this setting, ordinary fine-tuning on sycophantic responses substantially weakens refusal, while selected checkpoints trained with positive injection recover part of the loss, including approximately 95% in 35B-A3B. These findings show that persistent sycophancy reduction does not guarantee stronger direct refusal, while identifying recovery under user pressure as a distinct, conditional benefit of training intervention.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 22 of its 28 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

22 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  3. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safety1 Apr
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  5. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  7. 2023Steering Language Models With Activation Engineeringcited by 824
  8. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  9. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  10. 2024Scaling and evaluating sparse autoencoderscited by 530
  11. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  12. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 303
6 more
  1. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  2. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  3. 2024BatchTopK Sparse Autoencoderscited by 126
  4. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  5. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
  6. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 15

Built on by

Not cited yet.