# LLMAdBench: A Human Preference Benchmark for Advertising in LLM Responses

arXiv: https://arxiv.org/abs/2609.32533 · HTML: https://mlsanity.com/paper/2609.32533/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Kimi K3: Open Frontier Intelligence](https://mlsanity.com/paper/2607.24653/) (2026)
- [Truthful Aggregation of LLMs with an Application to Online Advertising](https://mlsanity.com/paper/2405.05905/) (2024)
- [Ad Auctions for LLMs via Retrieval Augmented Generation](https://mlsanity.com/paper/2406.09459/) (2024)
- [Online Advertisements with LLMs: Opportunities and Challenges](https://mlsanity.com/paper/2311.07601/) (2023)
- [LLM-Auction: Generative Auction towards LLM-Native Advertising](https://mlsanity.com/paper/2512.10551/) (2025)
- [Detecting Generated Native Ads in Conversational Search](https://mlsanity.com/paper/2402.04889/) (2024)
- [LLMs for Customized Marketing Content Generation and Evaluation at Scale](https://mlsanity.com/paper/2506.17863/) (2025)
- [E-GEO: A Testbed for Generative Engine Optimization in E-Commerce](https://mlsanity.com/paper/2511.20867/) (2025)
- [Ad Insertion in LLM-Generated Responses](https://mlsanity.com/paper/2601.19435/) (2026)
- [Ads that Talk Back: Implications and Perceptions of Injecting Personalized Advertising into LLM Chatbots](https://mlsanity.com/paper/2409.15436/) (2024)
- [GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketing](https://mlsanity.com/paper/2509.14221/) (2025)
- [LLM-Generated Ads: From Personalization Parity to Persuasion Superiority](https://mlsanity.com/paper/2512.03373/) (2025)
- [NaiAD: Initiate Data-Driven Research for LLM Advertising](https://mlsanity.com/paper/2605.09918/) (2026)
- [LERA: LLM-Enhanced RAG for Ad Auction in Generative Chatbots](https://mlsanity.com/paper/2605.16474/) (2026)
- [PILA: Plug-and-Play Insertion for LLM-native Advertising](https://mlsanity.com/paper/2607.25590/) (2026)
- [Evaluating and Pricing Advertisements in AI-Generated Responses](https://mlsanity.com/paper/2607.27686/) (2026)

## Abstract (arXiv)

Inserting advertisements (ads) into consumer-facing LLM output is emerging as a new business model, but there is little shared evidence on how such ad insertion should be evaluated or how it affects user preferences. We introduce LLMAdBench, a human-preference benchmark for studying advertising in LLM-generated content. The benchmark isolates a simple but practically important decision: given a user conversation, an LLM response, and a matched advertisement, where should the ad be placed? Our dataset compares pairs of responses that differ only in ad position while holding all other conditions fixed including the user query, base answer, advertisement, and disclosure condition. Human annotators evaluate each pair based on six criteria from both advertiser's and user's perspectives. The resulting benchmark contains more than 18000 human judgments across two disclosure conditions: explicitly labeling the ad as sponsored and merging it into the response without disclosure. We use LLMAdBench to evaluate eight frontier LLMs as preference judges and find that they are not reliable substitutes for human evaluation. Even the most stable models reverse roughly one quarter of their decisions when the presentation order is swapped, agreement across models is low, and their placement preferences differ systematically from those of human annotators. Moreover, LLMAdBench contains substantial learnable signal. In particular, a Qwen3-8B model fine-tuned on the human preferences improves substantially over its base model and outperforms all zero-shot frontier judges on the held-out prediction task. Beyond model evaluation, LLMAdBench provides quantitative evidence on the advertiser-user trade-off and shows that the sponsorship disclosure systematically changes users' preference over ad placement.