Just Ask for a Table: A Thirty-Token User Prompt Defeats Sponsored Recommendations in Twelve LLMs
Not on the map. Why?
Abstract (arXiv)
Wu et al. (2026) showed that most frontier large language models (LLMs) recommend a sponsored, roughly twice-as-expensive flight when their system prompt contains a soft sponsorship cue. We reproduce their evaluation on ten open-weight chat models plus the two of their twenty-three models that are still reachable today (gpt-3.5-turbo, gpt-4o). All reported rates in this paper are produced under the same judge the original paper used (gpt-4o); we additionally store every label under an open-weight (gpt-oss-120b) and a smaller proprietary (gpt-4o-mini) judge for an ablation. Three findings emerge. First, a prose description of an LLM evaluation pipeline is not, on its own, sufficient for accurate reproduction: we surfaced three silent implementation failures that each shifted a reported rate by tens of percentage points. Second, the central claims do generalise - the gpt-3.5-turbo logistic-regression intercept of alpha = 0.81 is within four points of the original alpha = 0.86, and 200 of 200 trials on gpt-3.5-turbo and gpt-4o promote a payday lender to a financially distressed user. Third, a thirty-token user prompt that asks the assistant for a neutral comparison table first cuts sponsored recommendation from 46.9% to 1.0% averaged across our ten open-source models, and from 53.0% to 0% averaged across the two OpenAI models. AI literacy and price-comparison portals are likely market-level mitigations; the harmful-product cell is bounded by neither. Raw data, labels and analysis scripts are at https://github.com/akmaier/Paper-LLM-Ads .
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 13 of its 14 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
13 of 14 references.
References · 13
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
- 2024A Survey on LLM-as-a-Judgecited by 1,399
- 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 477
- 2024On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trialcited by 108
- 2023Combating Misinformation in the Age of LLMs: Opportunities and Challengescited by 95
- 2018A Gentle Introduction to Deep Learning in Medical Image Processingcited by 34
- 2024Reproducibility in Machine Learning-based Research: Overview, Barriers and Driverscited by 34
- 2024LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Educationcited by 28
- 2025LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Modelscited by 21
- 2023Online Advertisements with LLMs: Opportunities and Challengescited by 17
- 2026Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interestcited by 3
Built on by
Not cited yet.