# PromptBench: A Unified Library for Evaluation of Large Language Models

arXiv: https://arxiv.org/abs/2312.07910 · HTML: https://mlsanity.com/paper/2312.07910/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2023.

## Built on by (most cited first, 30 of 44)

- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections](https://mlsanity.com/paper/2510.09023/) (2025)
- [Efficient multi-prompt evaluation of LLMs](https://mlsanity.com/paper/2405.17202/) (2024)
- [Benchmarks as Microscopes: A Call for Model Metrology](https://mlsanity.com/paper/2407.16711/) (2024)
- [Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping](https://mlsanity.com/paper/2410.08695/) (2024)
- [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026)
- [Enhancing LLM Robustness to Perturbed Instructions: An Empirical Study](https://mlsanity.com/paper/2504.02733/) (2025)
- [On Catastrophic Inheritance of Large Foundation Models](https://mlsanity.com/paper/2402.01909/) (2024)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models](https://mlsanity.com/paper/2409.12106/) (2024)
- [The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluators](https://mlsanity.com/paper/2406.12319/) (2024)
- [CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcare](https://mlsanity.com/paper/2512.11437/) (2025)
- [Behavior Injection: Preparing Language Models for Reinforcement Learning](https://mlsanity.com/paper/2505.18917/) (2025)
- [Safe-Child-LLM: A Developmental Benchmark for Evaluating LLM Safety in Child-LLM Interactions](https://mlsanity.com/paper/2506.13510/) (2025)
- [The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formatting](https://mlsanity.com/paper/2505.12837/) (2025)
- [Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators](https://mlsanity.com/paper/2505.14314/) (2025)
- [MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?](https://mlsanity.com/paper/2606.23664/) (2026)
- [Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions](https://mlsanity.com/paper/2508.13214/) (2025)
- [Semantic-Preserving Prompt Hijacking: A Black-Box Adversarial Attack on Auto-Prompt Optimization](https://mlsanity.com/paper/2506.18756/) (2025)
- [MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs](https://mlsanity.com/paper/2507.09701/) (2025)
- [Too Big to Fool: Resisting Deception in Language Models](https://mlsanity.com/paper/2412.10558/) (2024)
- [Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separators](https://mlsanity.com/paper/2504.05689/) (2025)
- [Test It Before You Trust It: Applying Software Testing for Trustworthy In-context Learning](https://mlsanity.com/paper/2504.18827/) (2025)
- [FLASH-D: FlashAttention with Hidden Softmax Division](https://mlsanity.com/paper/2505.14201/) (2025)
- [GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testing](https://mlsanity.com/paper/2507.07735/) (2025)
- [Syntactic Framing Fragility: An Audit of Robustness in LLM Ethical Decisions](https://mlsanity.com/paper/2601.09724/) (2025)
- [Output Scouting: Auditing Large Language Models for Catastrophic Responses](https://mlsanity.com/paper/2410.05305/) (2024)
- [TeleEval-OS: Performance evaluations of large language models for operations scheduling](https://mlsanity.com/paper/2506.11017/) (2025)
- [Re-Evaluating Code LLM Benchmarks Under Semantic Mutation](https://mlsanity.com/paper/2506.17369/) (2025)
- [SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Code](https://mlsanity.com/paper/2512.05954/) (2025)

## Abstract (arXiv)

The evaluation of large language models (LLMs) is crucial to assess their performance and mitigate potential security risks. In this paper, we introduce PromptBench, a unified library to evaluate LLMs. It consists of several key components that are easily used and extended by researchers: prompt construction, prompt engineering, dataset and model loading, adversarial prompt attack, dynamic evaluation protocols, and analysis tools. PromptBench is designed to be an open, general, and flexible codebase for research purposes that can facilitate original study in creating new benchmarks, deploying downstream applications, and designing new evaluation protocols. The code is available at: https://github.com/microsoft/promptbench and will be continuously supported.