mlsanityData to 7 Oct 2026

PromptBench: A Unified Library for Evaluation of Large Language Models

arXiv 2312.07910 ↗First posted 13 Dec 2023cs.AI cs.CL cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

The evaluation of large language models (LLMs) is crucial to assess their performance and mitigate potential security risks. In this paper, we introduce PromptBench, a unified library to evaluate LLMs. It consists of several key components that are easily used and extended by researchers: prompt construction, prompt engineering, dataset and model loading, adversarial prompt attack, dynamic evaluation protocols, and analysis tools. PromptBench is designed to be an open, general, and flexible codebase for research purposes that can facilitate original study in creating new benchmarks, deploying downstream applications, and designing new evaluation protocols. The code is available at: https://github.com/microsoft/promptbench and will be continuously supported.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Dec 2023.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 23; 2026: 10.

42 papers.

Sort:

By area:

  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  2. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  3. 2024Efficient multi-prompt evaluation of LLMscited by 72
  4. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  5. 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrappingcited by 22
  6. 2025Enhancing LLM Robustness to Perturbed Instructions: An Empirical Studycited by 17
  7. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 16
  8. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  9. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  10. 2024Measuring Human and AI Values Based on Generative Psychometrics with Large Language Modelscited by 13
  11. 2024The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluatorscited by 11
  12. 2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
30 more
  1. 2025CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcarecited by 7
  2. 2025The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formattingcited by 5
  3. 2025Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operatorscited by 5
  4. 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
  5. 2026MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?cited by 4
  6. 2025MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMscited by 3
  7. 2024Too Big to Fool: Resisting Deception in Language Modelscited by 2
  8. 2025Test It Before You Trust It: Applying Software Testing for Trustworthy In-context Learningcited by 2
  9. 2025FLASH-D: FlashAttention with Hidden Softmax Divisioncited by 2
  10. 2025Semantic-Preserving Prompt Hijacking: A Black-Box Adversarial Attack on Auto-Prompt Optimizationcited by 2
  11. 2025GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testingcited by 2
  12. 2025Syntactic Framing Fragility: An Audit of Robustness in LLM Ethical Decisionscited by 2
  13. 2024Output Scouting: Auditing Large Language Models for Catastrophic Responsescited by 1
  14. 2025Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separatorscited by 1
  15. 2025TeleEval-OS: Performance evaluations of large language models for operations schedulingcited by 1
  16. 2025Re-Evaluating Code LLM Benchmarks Under Semantic Mutationcited by 1
  17. 2026ReasonLab: A Controlled and Auditable Evaluation of Prompting Techniques for Multiple-Choice QAcited by 1
  18. 2026When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratingscited by 1
  19. 2026ZonoGPT: Towards An Abstract Domain for Verifying Large GPT Modelscited by 1
  20. 2025Provably Secure Retrieval-Augmented Generationcited by 0
  21. 2025Characterizing Fitness Landscape Structures in Prompt Engineeringcited by 0
  22. 2025Vectorized FlashAttention with Low-cost Exponential Computation in RISC-V Vector Processorscited by 0
  23. 2025SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Codecited by 0
  24. 2025Quantifying Return on Security Controls in LLM Systemscited by 0
  25. 2025CienaLLM: Generative Climate-Impact Extraction from News Articles with Autoregressive LLMscited by 0
  26. 2026Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarkscited by 0
  27. 2026Evaluating Prompting Strategies for Chart Question Answering with Large Language Modelscited by 0
  28. 2026An Interpretable and Scalable Framework for Evaluating Large Language Modelscited by 0
  29. 2026Measuring Behavior Portability in Large Language Modelscited by 0
  30. 2026aiXamine: Unified Black-Box Evaluation of Cross-Dimensional Trade-offs in LLM Safety, Security, and Privacycited by 0