# GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketing

arXiv: https://arxiv.org/abs/2509.14221 · HTML: https://mlsanity.com/paper/2509.14221/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 26 of 26)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [SmartPlay: A Benchmark for LLMs as Intelligent Agents](https://mlsanity.com/paper/2310.01557/) (2023)
- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2401.16745/) (2024)
- [GameArena: Evaluating LLM Reasoning through Live Computer Games](https://mlsanity.com/paper/2412.06394/) (2024)
- [Auction learning as a two-player game](https://mlsanity.com/paper/2006.05684/) (2020)
- [Auctions with LLM Summaries](https://mlsanity.com/paper/2404.08126/) (2024)
- [Ad Auctions for LLMs via Retrieval Augmented Generation](https://mlsanity.com/paper/2406.09459/) (2024)
- [Striking Gold in Advertising: Standardization and Exploration of Ad Text Generation](https://mlsanity.com/paper/2309.12030/) (2023)
- [Online Advertisements with LLMs: Opportunities and Challenges](https://mlsanity.com/paper/2311.07601/) (2023)
- [Scalable Semantic Matching of Queries to Ads in Sponsored Search Advertising](https://mlsanity.com/paper/1607.01869/) (2016)
- [Detecting Generated Native Ads in Conversational Search](https://mlsanity.com/paper/2402.04889/) (2024)
- [VALUE: Value-Aware Large Language Model for Query Rewriting via Weighted Trie in Sponsored Search](https://mlsanity.com/paper/2504.05321/) (2025)
- [AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising](https://mlsanity.com/paper/2408.05906/) (2024)

## Built on by (most cited first, 7 of 7)

- [LLM-Auction: Generative Auction towards LLM-Native Advertising](https://mlsanity.com/paper/2512.10551/) (2025)
- [NaiAD: Initiate Data-Driven Research for LLM Advertising](https://mlsanity.com/paper/2605.09918/) (2026)
- [Generative AI Advertising as a Problem of Trustworthy Commercial Intervention](https://mlsanity.com/paper/2605.18673/) (2026)
- [Evaluating and Pricing Advertisements in AI-Generated Responses](https://mlsanity.com/paper/2607.27686/) (2026)
- [AdsWorldEngine: A Self-Evolving Conversational Advertising Agent through Orchestrator and Tool Coevolution](https://mlsanity.com/paper/2608.13833/) (2026)
- [Token-Level Advertising](https://mlsanity.com/paper/2608.27382/) (2026)
- [LLMAdBench: A Human Preference Benchmark for Advertising in LLM Responses](https://mlsanity.com/paper/2609.32533/) (2026)

## Abstract (arXiv)

Generative Engine Marketing (GEM) is an emerging ecosystem for monetizing generative engines, such as LLM-based chatbots, by seamlessly integrating relevant advertisements into their responses. At the core of GEM lies the generation and evaluation of ad-injected responses. However, existing benchmarks are not specifically designed for this purpose, which limits future research. To address this gap, we propose GEM-Bench, the first comprehensive benchmark for ad-injected response generation in GEM. GEM-Bench includes three curated datasets covering both chatbot and search scenarios, a metric ontology that captures multiple dimensions of user satisfaction and engagement, and several baseline solutions implemented within an extensible multi-agent framework. Our preliminary results indicate that, while simple prompt-based methods achieve reasonable engagement such as click-through rate, they often reduce user satisfaction. In contrast, approaches that insert ads based on pre-generated ad-free responses help mitigate this issue but introduce additional overhead. These findings highlight the need for future research on designing more effective and efficient solutions for generating ad-injected responses in GEM. The benchmark and all related resources are publicly available at https://gem-bench.org/.