When Meaning Stays the Same, but Models Drift: Evaluating Quality of Service under Token-Level Behavioral Instability in LLMs
Not on the map. Why?
Abstract (arXiv)
We investigate how large language models respond to prompts that differ only in their token-level realization but preserve the same semantic intent, a phenomenon we call prompt variance. We propose Prompt-Based Semantic Shift (PBSS), a diagnostic framework for measuring behavioral drift in LLMs under semantically equivalent prompt rewordings. Applied to ten constrained tasks, PBSS reveals consistent, model-specific response shifts, suggesting statistical regularities linked to tokenization and decoding. These results highlight an overlooked dimension of model evaluation stability under rephrasing and suggest that tokenization strategies and decoding dynamics may contribute to post-training quality of service instability.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 12 of its 21 references that are in the map. First posted on arXiv 11 Jun 2025.
Builds on
12 of 21 references.
References · 12
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
- 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
- 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
- 2022Predictability and Surprise in Large Generative Modelscited by 210
- 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
- 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injectioncited by 59
- 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 42
- 2024Utilize the Flow before Stepping into the Same River Twice: Certainty Represented Knowledge Flow for Refusal-Aware Instruction Tuningcited by 10
Built on by
Citing papers per year: 2026: 2.
2 papers.