# LLMs for Customized Marketing Content Generation and Evaluation at Scale

arXiv: https://arxiv.org/abs/2506.17863 · HTML: https://mlsanity.com/paper/2506.17863/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 22 Jun 2025.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning](https://mlsanity.com/paper/2410.08146/) (2024)
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [Self-Discover: Large Language Models Self-Compose Reasoning Structures](https://mlsanity.com/paper/2402.03620/) (2024)
- [Copyright Violations and Large Language Models](https://mlsanity.com/paper/2310.13771/) (2023)
- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://mlsanity.com/paper/2310.14724/) (2023)
- [Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate](https://mlsanity.com/paper/2305.13160/) (2023)
- [Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation](https://mlsanity.com/paper/2402.09267/) (2024)
- [Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge](https://mlsanity.com/paper/2501.18099/) (2025)
- [Privacy Issues in Large Language Models: A Survey](https://mlsanity.com/paper/2312.06717/) (2023)
- [Controllable Text Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2408.12599/) (2024)
- [Disclosure and Mitigation of Gender Bias in LLMs](https://mlsanity.com/paper/2402.11190/) (2024)
- [Detecting and Mitigating Hallucinations in Multilingual Summarisation](https://mlsanity.com/paper/2305.13632/) (2023)
- [Can LLMs Follow Simple Rules?](https://mlsanity.com/paper/2311.04235/) (2023)
- [I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm](https://mlsanity.com/paper/2408.08072/) (2024)
- [IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering](https://mlsanity.com/paper/2408.13545/) (2024)
- [ReasoningRec: Bridging Personalized Recommendations and Human-Interpretable Explanations through LLM Reasoning](https://mlsanity.com/paper/2410.23180/) (2024)
- [GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checking](https://mlsanity.com/paper/2502.16514/) (2025)
- [SERP Interference Network and Its Applications in Search Advertising](https://mlsanity.com/paper/2506.21598/) (2025)

## Built on by (most cited first, 9 of 9)

- [Who Gets Which Message? Auditing Demographic Bias in LLM-Generated Targeted Text](https://mlsanity.com/paper/2601.17172/) (2026)
- [HLLM-Creator: Hierarchical LLM-based Personalized Creative Generation](https://mlsanity.com/paper/2508.18118/) (2025)
- [From Chains to Graphs: Self-Structured Reasoning for General-Domain LLMs](https://mlsanity.com/paper/2601.03597/) (2026)
- [Self-Reasoning Agentic Framework for Narrative Product Grid-Collage Generation](https://mlsanity.com/paper/2604.16958/) (2026)
- [Using item recommendations and LLMs in marketing email titles](https://mlsanity.com/paper/2508.20024/) (2025)
- [Recontextualizing Famous Quotes for Brand Slogan Generation](https://mlsanity.com/paper/2602.06049/) (2026)
- [Hybrid Intent-Aware Personalization with Machine Learning and RAG-Enabled Large Language Models for Financial Services Marketing](https://mlsanity.com/paper/2603.14173/) (2026)
- [Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning](https://mlsanity.com/paper/2607.16057/) (2026)
- [LLMAdBench: A Human Preference Benchmark for Advertising in LLM Responses](https://mlsanity.com/paper/2609.32533/) (2026)

## Abstract (arXiv)

Offsite marketing is essential in e-commerce, enabling businesses to reach customers through external platforms and drive traffic to retail websites. However, most current offsite marketing content is overly generic, template-based, and poorly aligned with landing pages, limiting its effectiveness. To address these limitations, we propose MarketingFM, a retrieval-augmented system that integrates multiple data sources to generate keyword-specific ad copy with minimal human intervention. We validate MarketingFM via offline human and automated evaluations and large-scale online A/B tests. In one experiment, keyword-focused ad copy outperformed templates, achieving up to 9% higher CTR, 12% more impressions, and 0.38% lower CPC, demonstrating gains in ad ranking and cost efficiency. Despite these gains, human review of generated ads remains costly. To address this, we propose AutoEval-Main, an automated evaluation system that combines rule-based metrics with LLM-as-a-Judge techniques to ensure alignment with marketing principles. In experiments with large-scale human annotations, AutoEval-Main achieved 89.57% agreement with human reviewers. Building on this, we propose AutoEval-Update, a cost-efficient LLM-human collaborative framework to dynamically refine evaluation prompts and adapt to shifting criteria with minimal human input. By selectively sampling representative ads for human review and using a critic LLM to generate alignment reports, AutoEval-Update improves evaluation consistency while reducing manual effort. Experiments show the critic LLM suggests meaningful refinements, improving LLM-human agreement. Nonetheless, human oversight remains essential for setting thresholds and validating refinements before deployment.