mlsanityData to 7 Oct 2026

Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations

arXiv 2403.06009 ↗First posted 9 Mar 2024cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are susceptible to a variety of risks, from non-faithful output to biased and toxic generations. Due to several limiting factors surrounding LLMs (training cost, API access, data availability, etc.), it may not always be feasible to impose direct safety constraints on a deployed model. Therefore, an efficient and reliable alternative is required. To this end, we present our ongoing efforts to create and deploy a library of detectors: compact and easy-to-build classification models that provide labels for various harms. In addition to the detectors themselves, we discuss a wide range of uses for these detector models - from acting as guardrails to enabling effective AI governance. We also deep dive into inherent challenges in their development and discuss future work aimed at making the detectors more reliable and broadening their scope.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 73 of its 155 references that are in the map. First posted on arXiv 9 Mar 2024.

Builds on

73 of 155 references.

References · 73

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  8. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  9. 2023Mistral 7Bcited by 3,065
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
38 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2022Red Teaming Language Models with Language Modelscited by 983
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  6. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  7. 2022A Survey on In-context Learningcited by 717
  8. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  9. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  10. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  13. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  14. 2023Detecting Language Model Attacks with Perplexitycited by 411
  15. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  16. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  17. 2020Classification with Valid and Adaptive Coveragecited by 390
  18. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  19. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  20. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  21. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  22. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  23. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  24. 2022Generating Sequences by Learning to Self-Correctcited by 271
  25. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  26. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  27. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  28. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  29. 2023RADAR: Robust AI-Text Detection via Adversarial Learningcited by 187
  30. 2021Challenges in Detoxifying Language Modelscited by 169
  31. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  32. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
  33. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  34. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  35. 2020Uncertainty Quantification and Deep Ensemblescited by 138
  36. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
  37. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  38. 2020Generate Your Counterfactuals: Towards Controlled Counterfactual Generation for Textcited by 75

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 6; 2025: 4.

10 papers.

  1. 2024Grade Like a Human: Rethinking Automated Assessment with Large Language Modelscited by 16
  2. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  3. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  4. 2025ASTRAL: Automated Safety Testing of Large Language Modelscited by 7
  5. 2024Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forwardcited by 5
  6. 2025Understanding and Mitigating Risks of Generative AI in Financial Servicescited by 4
  7. 2024When in Doubt, Cascade: Towards Building Efficient and Capable Guardrailscited by 1
  8. 2025Fine-Tuning Large Language Models with QLoRA for Offensive Language Detection in Roman Urdu-English Code-Mixed Textcited by 1
  9. 2024Trading Devil Final: Backdoor attack via Stock market and Bayesian Optimizationcited by 0
  10. 2025A Library of LLM Intrinsics for Retrieval-Augmented Generationcited by 0