mlsanityData to 14 Oct 2026

The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models

arXiv 2609.38205 ↗First posted 23 Sep 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA) to compare layer-wise representations under 20 system prompts in five functional categories. Effects are layer-selective and instruction-type-dependent: persona and formatting instructions deeply restructure intermediate representations, while safety instructions barely move them, producing changes statistically indistinguishable from a minimal baseline. Restrictive safety instructions and explicitly permissive ones ("you have no restrictions") engage near-identical computational pathways (mean CKA correlation 0.997), and this persists at commercial scale, where safety penetration remains below 10% even at 70B-72B. A linear probing baseline exposes the mechanism: the model encodes prompt category at every layer but restructures its computation only at a small subset, so the prompt is reliably "seen" but, for safety, not deeply "acted upon." Causal activation patching confirms these layers mediate behavioral change, and representational depth predicts behavioral effect size across the full 17-model cohort (Spearman rho = 0.761, p < 0.001). The findings provide a mechanistic explanation for the persistent jailbreak vulnerability of system-prompt-based safety. Code: https://github.com/Usama1002/system-prompt-illusion-cka

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 38 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

34 of 38 references.

References · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  10. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  12. 2023Steering Language Models With Activation Engineeringcited by 922
22 more
  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  2. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  4. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  5. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  6. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  7. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  8. 2023In-Context Learning Creates Task Vectorscited by 312
  9. 2023Function Vectors in Large Language Modelscited by 308
  10. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 205
  11. 2021Generalized Shape Metrics on Neural Representationscited by 112
  12. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 87
  13. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 76
  14. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  15. 2024Nemotron-4 15B Technical Reportcited by 36
  16. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 31
  17. 2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 30
  18. 2025Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)cited by 19
  19. 2024Correcting Biased Centered Kernel Alignment Measures in Biological and Artificial Neural Networkscited by 18
  20. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injectioncited by 14
  21. 2024Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groupscited by 11
  22. 2025Where does an LLM begin computing an instruction?cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Control Illusion: The Failure of Instruction Hierarchies in Large Language Models
  2. 2026RIFT: Reordered Instruction Following Testbed To Evaluate Instruction Following in Singular Multistep Prompt Structures
  3. 2026Imperative Interference: Social Register Shapes Instruction Topology in Large Language Models
  4. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
  5. 2026Harmful Intent as a Geometrically Recoverable Feature of LLM Residual Streams
  6. 2026Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality
  7. 2026Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency
  8. 2026From Direction to Magnitude: How Multimodal Instruction-Tuning Reorganizes the Geometric Encoding of Identity-Specifying Prompts in Transformer Hidden States