Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs
On the map: Mixture-of-experts → Mixture of LoRA experts
Abstract (arXiv)
Prompt-based adaptation offers several ways to specialize a frozen language model, ranging from natural-language prompt optimization to learned continuous prompts. These methods are usually compared through downstream performance, although similar outputs need not arise from similar internal computations. We study how discrete prompt optimization, prompt tuning, and prefix tuning alter the processing of dense and mixture-of-experts language models. Our analysis shows that the location of the adaptation predicts how task information is carried to the output: optimized text primarily changes response construction, input-level prompts write their influence into content representations, and layer-wise prefixes maintain direct control through attention. In MoE models, these differences further induce distinct expert-routing patterns. At the same time, common representation-level measurements respond strongly to generic context changes and do not reliably distinguish useful adaptations from matched placebos. We argue that understanding prompt adaptation requires separating representation drift from the part of that drift that is aligned with, and causally used by, the model's task computation. This perspective provides practical guidance for selecting, evaluating, and interpreting prompt adaptation methods.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 42 of its 49 references that are in the map. First posted on arXiv 13 Oct 2026.
Builds on
42 of 49 references.
References · 42
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
- 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
- 2023Steering Language Models With Activation Engineeringcited by 922
- 2023Large Language Models as Optimizerscited by 864
- 2022Transformers learn in-context by gradient descentcited by 732
- 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
- 2019Designing and Interpreting Probes with Control Taskscited by 640
30 more
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
- 2024Scaling and evaluating sparse autoencoderscited by 553
- 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
- 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
- 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
- 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
- 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
- 2023In-Context Learning Creates Task Vectorscited by 312
- 2023Function Vectors in Large Language Modelscited by 308
- 2024Massive Activations in Large Language Modelscited by 294
- 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
- 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
- 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
- 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
- 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
- 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
- 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 52
- 2023On the Role of Attention in Prompt-tuningcited by 50
- 2023When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitationscited by 43
- 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 29
- 2025The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Modelscited by 18
- 2025Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsitycited by 18
- 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
- 2025Probing Semantic Routing in Large Mixture-of-Expert Modelscited by 14
- 2025Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modulescited by 8
- 2026Steer Like the LLM: Activation Steering that Mimics Promptingcited by 5
- 2024Eliciting Textual Descriptions from Representations of Continuous Promptscited by 4
- 2025Towards Interpretable Soft Promptscited by 4
- 2026Hierarchical Mixture-of-Experts with Two-Stage Optimizationcited by 3
- 2026Decomposing how prompting steers behaviorcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025PromptFlow: Training Prompts Like Neural Networks
- 2026Learning an Anchored Prompt Space for Continual Adaptation of Large Language Models
- 2026Why Prompt Optimization Works, and Why It Sometimes Doesn't: A Causal-Inspired Edit-Level Analysis
- 2024On the Transformations across Reward Model, Parameter Update, and In-Context Prompt
- 2024The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting
- 2026HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts
- 2026Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality
- 2025Bias after Prompting: Persistent Discrimination in Large Language Models