# AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising

arXiv: https://arxiv.org/abs/2408.05906 · HTML: https://mlsanity.com/paper/2408.05906/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Click-Through Rate Prediction in Online Advertising: A Literature Review](https://mlsanity.com/paper/2202.10462/) (2022)
- [Learning to Create Better Ads: Generation and Ranking Approaches for Ad Creative Refinement](https://mlsanity.com/paper/2008.07467/) (2020)
- [Striking Gold in Advertising: Standardization and Exploration of Ad Text Generation](https://mlsanity.com/paper/2309.12030/) (2023)
- [DeepGen: Diverse Search Ad Generation and Real-Time Customization](https://mlsanity.com/paper/2208.03438/) (2022)
- [Natural Language Generation for Advertising: A Survey](https://mlsanity.com/paper/2306.12719/) (2023)
- [On Evaluating Multilingual Compositional Generalization with Translated Datasets](https://mlsanity.com/paper/2306.11420/) (2023)
- [Aspect-based Analysis of Advertising Appeals for Search Engine Advertising](https://mlsanity.com/paper/2204.11445/) (2022)
- [Linguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgments](https://mlsanity.com/paper/1901.03438/) (2019)

## Built on by (most cited first, 4 of 4)

- [GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketing](https://mlsanity.com/paper/2509.14221/) (2025)
- [RELATE: A Reinforcement Learning-Enhanced LLM Framework for Advertising Text Generation](https://mlsanity.com/paper/2602.11780/) (2026)
- [NaiAD: Initiate Data-Driven Research for LLM Advertising](https://mlsanity.com/paper/2605.09918/) (2026)
- [Exploring the Relationship Between Diversity and Quality in Ad Text Generation](https://mlsanity.com/paper/2505.16418/) (2025)

## Abstract (arXiv)

With the increase in the fluency of ad texts automatically created by natural language generation technology, there is high demand to verify the quality of these creatives in a real-world setting. We propose AdTEC (Ad Text Evaluation Benchmark by CyberAgent), the first public benchmark to evaluate ad texts from multiple perspectives within practical advertising operations. Our contributions are as follows: (i) Defining five tasks for evaluating the quality of ad texts, as well as building a Japanese dataset based on the practical operational experiences of building a Japanese dataset based on the practical operational experiences of advertising agencies, which are typically kept in-house. (ii) Validating the performance of existing pre-trained language models (PLMs) and human evaluators on the dataset. (iii) Analyzing the characteristics and providing challenges of the benchmark. The results show that while PLMs have already reached practical usage level in several tasks, humans still outperform in certain domains, implying that there is significant room for improvement in this area.