Behavioral Steering in a 35B MoE Language Model via SAE-Decoded Probe Vectors: One Agency Axis, Not Five Traits
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
also in 1 more
Abstract (arXiv)
We train nine sparse autoencoders (SAEs) on the residual stream of Qwen 3.5-35B-A3B, a 35-billion-parameter Mixture-of-Experts model with a hybrid GatedDeltaNet/attention architecture, and use them to identify and steer five agentic behavioral traits. Our method trains linear probes on SAE latent activations, then projects the probe weights back through the SAE decoder to obtain continuous steering vectors in the model's native activation space. This bypasses the SAE's top-k discretization, enabling fine-grained behavioral intervention at inference time with no retraining. Across 1,800 agent rollouts (50 scenarios times 36 conditions), we find that autonomy steering at multiplier 2 achieves Cohen's d = 1.01 (p < 0.0001), shifting the model from asking the user for help 78% of the time to proactively executing code and searching the web. Cross-trait analysis, however, reveals that all five steering vectors primarily modulate a single dominant agency axis (the disposition to act independently versus defer to the user), with trait specific effects appearing only as secondary modulations in tool-type composition and dose-response shape. The tool-use vector steers behavior (d = 0.39); the risk-calibration vector produces only suppression. We additionally show that steering only during autoregressive decoding has zero effect (p > 0.35), providing causal evidence that behavioral commitments are computed during prefill in GatedDeltaNet architectures.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 9 of its 13 references that are in the map. First posted on arXiv 17 Mar 2026.
Builds on
9 of 13 references.
References · 9
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
- 2024Scaling and evaluating sparse autoencoderscited by 530
- 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
- 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026WriteSAE: Sparse Autoencoders for Recurrent Statecited by 1
- 2026WriteSAE: Sparse Autoencoders for Recurrent State12 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026A Coin Flip Per Token: Bernoulli Sparse Steering of Large Language Models
- 2025A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering
- 2025Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atoms
- 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
- 2024Controlling Large Language Model Agents with Entropic Activation Steering
- 2026CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits
- 2026AutoTrainess: Teaching Language Models to Improve Language Models Autonomously
- 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models