Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation: Reproducibility Below the Rerun-Stability Baseline
Not on the map. Why?
Abstract (arXiv)
Small changes to how a buyer phrases a question -- "best CRM" vs "top CRM" vs "best CRM for a SaaS startup" -- produce substantially different brand recommendations from AI assistants. Across ~6,000 paraphrase runs and ~6,000 same-prompt rerun controls on OpenAI and Anthropic models, the recommendation-set similarity (Jaccard) between two paraphrases of the same underlying buying intent is 0.288 for cosmetic rewordings (clustered 95% CI [0.215, 0.361]) and 0.135 for constraint-adding rewordings ([0.098, 0.175], pooling region/language and specificity-ladder axes) -- both far below the 0.50-0.61 same-prompt rerun baseline. The prompt string, not the underlying buyer intent, is the dominant input to which brands surface. Increasing reasoning effort does not narrow the gap (bounded by +/-0.05). This is a direct challenge to an increasingly popular AEO/GEO practice. Tracking a brand's "AI visibility" by counting brand mentions over a fixed set of prompts produces a metric whose dominant source of variance is which paraphrase the tracker happens to issue, not the model's behavior toward the brand: the same buyer intent in two natural paraphrases produces recommendation sets that overlap 14-29% in Jaccard versus 50-61% for same-prompt reruns. Sampling more paraphrases per intent reduces the artifact in principle, and efficient multi-prompt evaluation methods exist in the academic literature, but the natural buyer-phrasing space is much larger than the benchmark-scale prompt sets those methods have been validated on, and far beyond what any commercial tracker issues per brand-intent combination. Prompt-by-prompt mention tracking is therefore structurally unstable as a unit of measurement; meaningful improvement likely requires a different unit rather than a larger prompt set.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 10 of its 16 references that are in the map. First posted on arXiv 22 May 2026.
Builds on
10 of 16 references.
References · 10
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
- 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 213
- 2024Efficient multi-prompt evaluation of LLMscited by 73
- 2024POSIX: A Prompt Sensitivity Index For Large Language Modelscited by 59
- 2023GEO: Generative Engine Optimizationcited by 56
- 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 53
- 2025Prompting Science Report 2: The Decreasing Value of Chain of Thought in Promptingcited by 13
- 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Levelcited by 11
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Prominence-Stratified Failure Modes in Retrieval-Augmented Commercial Recommendation: A 37,000-Run Audit
- 2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
- 2025AdParaphrase v2.0: Generating Attractive Ad Texts Using a Preference-Annotated Paraphrase Dataset
- 2025Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data
- 2025RoParQ: Paraphrase-Aware Alignment of Large Language Models Towards Robustness to Paraphrased Questions
- 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts
- 2024How often are errors in natural language reasoning due to paraphrastic variability?
- 2024Towards Human Understanding of Paraphrase Types in Large Language Models