As X, Do Y: How Persona and Task Combine in Instruction-Tuned LLMs
Not on the map. Why?
Abstract (arXiv)
Role prompts of the form As X, do Y admit a clean linear decomposition at one specific site in the residual stream: the prompt-to-answer transition -- the last prompt token together with the first two generated tokens -- in an early/mid layer band. There, persona and task contribute through partially orthogonal additive directions. Forming a pure persona effect DeltaX, a pure task effect DeltaY, and substituting hBB + DeltaX + DeltaY for the clean residual yields downstream output within a small KL of clean on Gemma-2-2B-IT and Qwen-2.5-1.5B, 3B-Instruct, across a 12-cell short grid and a 48-cell long-persona grid, with persona-specific behavioral markers preserved. The natural inference from this additive structure is that the role prompt can be compressed into a single cached residual vector. We show it cannot. Injecting the cached additive prediction -- or even the oracle clean residual hXY -- into a baseline host prompt with the persona text removed does not approach the clean long-persona target, at one site or at many layers. Persona-conditioned multi-token generation flows through attention back to the persona-text positions throughout the prompt, which no residual at one site reproduces. Local additivity in the residual stream does not imply prompt compressibility. The additive structure at the prompt-to-answer transition supports interpretability and fine-grained steering of persona or task contributions; persona-conditioned behavior across the full continuation depends on a distributed prompt/KV mechanism that local activation arithmetic does not displace.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 8 of its 14 references that are in the map. First posted on arXiv 22 May 2026.
Builds on
8 of 14 references.
References · 8
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,785
- 2022Locating and Editing Factual Associations in GPTcited by 2,650
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,286
- 2022Editing Models with Task Arithmeticcited by 1,101
- 2023In-Context Learning Creates Task Vectorscited by 305
- 2023Function Vectors in Large Language Modelscited by 299
- 2023Linearity of Relation Decoding in Transformer Language Modelscited by 195
Built on by
Not cited yet.