mlsanityData to 14 Oct 2026

Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality

arXiv 2608.20349 ↗First posted 16 Jun 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) exhibit extreme sensitivity to surface-level prompt variations, in which minor lexical changes can trigger disproportionate performance fluctuations. Moving beyond black-box optimization and coarse-grained templates, we present the first large-scale, n-gram token-level mechanistic analysis of prompt stability, leveraging a dataset of 132,000 prompt variants. Our investigation reveals a fundamental Scaling Law of Prompt Performance Stability: higher average task performance is strongly associated with lower variance and greater robustness across prompt perturbation. We identify two core linguistic drivers underlying this robustness: (1) Domain-Specific Terminology, which tightly anchors semantic boundaries, and (2) Explicit Action Directives, which formalize reasoning trajectories. Together, these elements constrain the model's interpretative space, effectively ``locking in'' more deterministic generation behavior. Building on these insights, we introduce an automated Prompt-Refining Agent that systematically restructures input queries by injecting domain anchoring and operational constraints. Empirical evaluation shows that our approach reduces performance variance by 40.7% in code generation task, while preserving or improving mean performance. These findings provide a statistically grounded and mechanistically interpretable framework for achieving robust prompt engineering.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

22 of 31 references.

References · 22

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  9. 2022A Survey on In-context Learningcited by 721
  10. 2021GPT Understands, Toocited by 716
  11. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  12. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
10 more
  1. 2021True Few-Shot Learning with Language Modelscited by 344
  2. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  3. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  4. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  5. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  6. 2024Aligning Language Models to Explicitly Handle Ambiguitycited by 56
  7. 2023How are Prompts Different in Terms of Sensitivity?cited by 35
  8. 2024Prompt Perturbation Consistency Learning for Robust Language Modelscited by 26
  9. 2025Towards LLMs Robustness to Changes in Prompt Format Stylescited by 21
  10. 2025Evaluating Robustness of Large Language Models Against Multilingual Typographical Errorscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompt Injection as an Emerging Threat: Evaluating the Resilience of Large Language Models
  2. 2025Improving Complex Reasoning with Dynamic Prompt Corruption: A soft prompt Optimization Approach
  3. 2025Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages
  4. 2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
  5. 2026Reflective Prompt Tuning through Language Model Function-Calling
  6. 2026Prompt Minimization: Reducing Input Redundancy Without Sacrificing Output Fidelity
  7. 2023Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis
  8. 2025PromptBridge: Cross-Model Prompt Transfer for Large Language Models