# Dynamic Evaluation for Oversensitivity in LLMs

arXiv: https://arxiv.org/abs/2510.19005 · HTML: https://mlsanity.com/paper/2510.19005/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Builds on (most cited first, 6 of 6)

- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks](https://mlsanity.com/paper/2309.17167/) (2023)
- [Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation](https://mlsanity.com/paper/2410.03415/) (2024)
- [Think Before Refusal : Triggering Safety Reflection in LLMs to Mitigate False Refusal Behavior](https://mlsanity.com/paper/2503.17882/) (2025)
- [PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routing](https://mlsanity.com/paper/2407.16318/) (2024)

## Built on by (most cited first, 2 of 2)

- [Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations](https://mlsanity.com/paper/2604.27093/) (2026)
- [WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses](https://mlsanity.com/paper/2609.36635/) (2026)

## Abstract (arXiv)

Oversensitivity occurs when language models defensively reject prompts that are actually benign. This behavior not only disrupts user interactions but also obscures the boundary between harmful and harmless content. Existing benchmarks rely on static datasets that degrade overtime as models evolve, leading to data contamination and diminished evaluative power. To address this, we develop a framework that dynamically generates model-specific challenging datasets, capturing emerging defensive patterns and aligning with each model's unique behavior. Building on this approach, we construct OVERBENCH, a benchmark that aggregates these datasets across diverse LLM families, encompassing 450,000 samples from 25 models. OVERBENCH provides a dynamic and evolving perspective on oversensitivity, allowing for continuous monitoring of defensive triggers as models advance, highlighting vulnerabilities that static datasets overlook.