ASSERT: A Measurement Pipeline for GenAI Audits
Not on the map. Why?
Abstract (arXiv)
Audits of generative AI (GenAI) systems often summarize behavior as a reported rate: how often the audited system complies with policy. Researchers and stakeholders use that rate to compare systems, track regressions, and gate deployment. A reported rate reflects both the system under audit and the measurement choices behind it, so a change in the rate can leave it unclear whether the system or those choices moved. We introduce ASSERT, a specification-driven measurement pipeline for GenAI audits that ties each reported rate to a written specification of the measurement choices used to produce it. ASSERT helps draft a behavioral rubric and test cases, then runs the audit against a GenAI system and returns a reported rate. In a case study on conversational deception, we observe that the reported rate moves substantially with the dialogue setup, the simulated user, the judge, and the evidence bar for non-compliance. These measurement choices substantially change the reported rate and can reorder GenAI system rankings. Because each reported rate is tied to an explicit specification, differences across audits are easier to attribute and interpret.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 56 references that are in the map. First posted on arXiv 14 Aug 2026.
Builds on
30 of 56 references.
References · 30
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
- 2018Datasheets for Datasetscited by 1,386
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2018Model Cards for Model Reportingcited by 1,080
- 2022Red Teaming Language Models with Language Modelscited by 1,026
- 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
- 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
- 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
18 more
- 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
- 2019Measurement and Fairnesscited by 308
- 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
- 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
- 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
- 2025Auditing language models for hidden objectivescited by 83
- 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 74
- 2019Veridical Data Sciencecited by 66
- 2025Position: Evaluating Generative AI Systems Is a Social Science Measurement Challengecited by 63
- 2025Measurement to Meaning: A Validity-Centered Framework for AI Evaluationcited by 57
- 2024Can You Trust LLM Judgments? Reliability of LLM-as-a-Judgecited by 56
- 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
- 2025Validating LLM-as-a-Judge Systems under Rating Indeterminacycited by 25
- 2024A Shared Standard for Valid Measurement of Generative AI Systems' Capabilities, Risks, and Impactscited by 13
- 2023One Model Many Scores: Using Multiverse Analysis to Prevent Fairness Hacking and Evaluate the Influence of Model Design Decisionscited by 12
- 2024BenchmarkCards: Standardized Documentation for Large Language Model Benchmarkscited by 10
- 2026Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiversecited by 7
- 2026AI-Assisted Systematization for Evaluating GenAI Systemscited by 1
Built on by
Not cited yet.