mlsanityData to 7 Oct 2026

Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems

arXiv 2401.05778 ↗First posted 11 Jan 2024cs.CL cs.AIcited by 63

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have strong capabilities in solving diverse natural language processing tasks. However, the safety and security issues of LLM systems have become the major obstacle to their widespread application. Many studies have extensively investigated risks in LLM systems and developed the corresponding mitigation strategies. Leading-edge enterprises such as OpenAI, Google, Meta, and Anthropic have also made lots of efforts on responsible LLMs. Therefore, there is a growing need to organize the existing studies and establish comprehensive taxonomies for the community. In this paper, we delve into four essential modules of an LLM system, including an input module for receiving prompts, a language model trained on extensive corpora, a toolchain module for development and deployment, and an output module for exporting LLM-generated content. Based on this, we propose a comprehensive taxonomy, which systematically analyzes potential risks associated with each module of an LLM system and discusses the corresponding mitigation strategies. Furthermore, we review prevalent benchmarks, aiming to facilitate the risk assessment of LLM systems. We hope that this paper can help LLM participants embrace a systematic perspective to build their responsible LLM systems.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 274 of its 417 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

274 of 417 references.

References · 274

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  12. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2016Deep Learning with Differential Privacycited by 2,682
  13. 2023A Survey of Large Language Modelscited by 2,537
  14. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  15. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  16. 2020Extracting Training Data from Large Language Modelscited by 2,144
  17. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  18. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  19. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  20. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  21. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  22. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  23. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  24. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  25. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  26. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  27. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  28. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  29. 2018Hierarchical Neural Story Generationcited by 1,182
  30. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  31. 2023PaLM 2 Technical Reportcited by 1,154
  32. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  33. 2023LIMA: Less Is More for Alignmentcited by 1,080
  34. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  35. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  36. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  37. 2022In-context Learning and Induction Headscited by 930
  38. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911

50 most cited of 274 All 274 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 29; 2026: 7.

63 papers.

Sort:

By area:

  1. 2024Tool Learning with Large Language Models: A Surveycited by 221
  2. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  3. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  5. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
  6. 2024DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learningcited by 52
  7. 2024Instruction Backdoor Attacks Against Customized LLMscited by 50
  8. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  9. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  10. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  11. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 40
  12. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
38 more
  1. 2024Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settingscited by 33
  2. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 27
  3. 2024Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?cited by 23
  4. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
  5. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 18
  6. 2025AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Modelscited by 16
  7. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  8. 2024Legilimens: Practical and Unified Content Moderation for Large Language Model Servicescited by 13
  9. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  10. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  11. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  12. 2025Malicious LLM-Based Conversational AI Makes Users Reveal Personal Informationcited by 10
  13. 2025Exposing LLM User Privacy via Traffic Fingerprint Analysis: A Study of Privacy Risks in LLM Agent Interactionscited by 9
  14. 2025Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applicationscited by 9
  15. 2024Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposalcited by 8
  16. 2024SafeAligner: Safety Alignment against Jailbreak Attacks via Response Disparity Guidancecited by 8
  17. 2024GenTel-Safe: A Unified Benchmark and Shielding Framework for Defending Against Prompt Injection Attackscited by 7
  18. 2024DocEdit-v2: Document Structure Editing Via Multimodal LLM Groundingcited by 7
  19. 2025Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generationcited by 7
  20. 2024SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibrationcited by 6
  21. 2024Enhancing Training Data Attribution for Large Language Models with Fitting Error Considerationcited by 6
  22. 2025What's Not Said Still Hurts: A Description-Based Evaluation Framework for Measuring Social Bias in LLMscited by 6
  23. 2025Invisible Filters: Cultural Bias in Hiring Evaluations Using Large Language Modelscited by 6
  24. 2026SHE: Trajectory-driven Safety Harness Evolution for LLM Agentscited by 6
  25. 2025A Survey on Proactive Defense Strategies Against Misinformation in Large Language Modelscited by 5
  26. 2025Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agentscited by 4
  27. 2025Bias Association Discovery Framework for Open-Ended LLM Generationscited by 4
  28. 2025MUSE: MCTS-Driven Red Teaming Framework for Enhanced Multi-Turn Dialogue Safety in Large Language Modelscited by 4
  29. 2024OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Datacited by 3
  30. 2024Safe-Embed: Unveiling the Safety-Critical Knowledge of Sentence Encoderscited by 3
  31. 2024ABC Align: Large Language Model Alignment for Safety & Accuracycited by 3
  32. 2025ChartLens: Fine-grained Visual Attribution in Chartscited by 3
  33. 2024CRiskEval: A Chinese Multi-Level Risk Evaluation Benchmark Dataset for Large Language Modelscited by 2
  34. 2024CTISum: A New Benchmark Dataset For Cyber Threat Intelligence Summarizationcited by 2
  35. 2025Securing Educational LLMs: A Generalised Taxonomy of Attacks on LLMs and DREAD Risk Assessmentcited by 2
  36. 2026Agent-Supported Foresight for AI Systemic Risks: AI Agents for Breadth, Experts for Judgmentcited by 2
  37. 2025PD3F: A Pluggable and Dynamic DoS-Defense Framework Against Resource Consumption Attacks Targeting Large Language Modelscited by 1
  38. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1

50 most cited of 63 All 63 (JSON)