mlsanityData to 14 Oct 2026

Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework

arXiv 2505.03563 ↗First posted 6 May 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are highly sensitive to subtle changes in prompt phrasing, posing challenges for reliable auditing. Prior methods often apply unconstrained prompt paraphrasing, which risk missing linguistic and demographic factors that shape authentic user interactions. We introduce AUGMENT (Automated User-Grounded Modeling and Evaluation of Natural Language Transformations), a framework for generating controlled paraphrases, grounded in user behaviors. AUGMENT leverages linguistically informed rules and enforces quality through checks on instruction adherence, semantic similarity, and realism, ensuring paraphrases are both reliable and meaningful for auditing. Through case studies on the BBQ and MMLU datasets, we show that controlled paraphrases uncover systematic weaknesses that remain obscured under unconstrained variation. These results highlight the value of the AUGMENT framework for reliable auditing.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

22 of 45 references.

References · 22

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  4. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  6. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  8. 2018Gender Bias in Coreference Resolutioncited by 442
  9. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  10. 2023Auditing large language models: a three-layered approachcited by 137
  11. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  12. 2023KoBBQ: Korean Bias Benchmark for Question Answeringcited by 71
10 more
  1. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 53
  2. 2020Evaluating Conversational Recommender Systems via User Simulationcited by 47
  3. 2024AI auditing: The Broken Bus on the Road to AI Accountabilitycited by 43
  4. 2024Social Bias Evaluation for Large Language Models Requires Prompt Variationscited by 40
  5. 2021Reliability Testing for Natural Language Processing Systemscited by 25
  6. 2024Paraphrase Types Elicit Prompt Engineering Capabilitiescited by 15
  7. 2022Detecting Text Formality: A Study of Text Classification Approachescited by 11
  8. 2024Why Don't Prompt-Based Fairness Metrics Correlate?cited by 4
  9. 2024Towards Human Understanding of Paraphrase Types in Large Language Modelscited by 2
  10. 2024Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairnesscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Intersectional Fairness in Large Language Modelscited by 0
  2. 2026Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibilitycited by 0
  3. 2026Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillationcited by 0