# A Use-Case Specific Dataset for Measuring Dimensions of Responsible Performance in LLM-generated Text

arXiv: https://arxiv.org/abs/2510.20782 · HTML: https://mlsanity.com/paper/2510.20782/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 23 Oct 2025.

## Builds on (most cited first, 9 of 9)

- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [Analyzing the Targets of Hate in Online Social Media](https://mlsanity.com/paper/1603.07709/) (2016)
- [Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semantics](https://mlsanity.com/paper/2206.03390/) (2022)
- [MetaHate: A Dataset for Unifying Efforts on Hate Speech Detection](https://mlsanity.com/paper/2401.06526/) (2024)
- [FAIR Enough: How Can We Develop and Assess a FAIR-Compliant Dataset for Large Language Models' Training?](https://mlsanity.com/paper/2401.11033/) (2024)

## Abstract (arXiv)

Current methods for evaluating large language models (LLMs) typically focus on high-level tasks such as text generation, without targeting a particular AI application. This approach is not sufficient for evaluating LLMs for Responsible AI dimensions like fairness, since protected attributes that are highly relevant in one application may be less relevant in another. In this work, we construct a dataset that is driven by a real-world application (generate a plain-text product description, given a list of product features), parameterized by fairness attributes intersected with gendered adjectives and product categories, yielding a rich set of labeled prompts. We show how to use the data to identify quality, veracity, safety, and fairness gaps in LLMs, contributing a proposal for LLM evaluation paired with a concrete resource for the research community.