mlsanityData to 7 Oct 2026

TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law

arXiv 2507.21134 ↗First posted 22 Jul 2025cs.CL cs.CY cs.LGcited by 7

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

As large language models (LLMs) are increasingly deployed in high-risk domains such as law, finance, and medicine, systematically evaluating their domain-specific safety and compliance becomes critical. While prior work has largely focused on improving LLM performance in these domains, it has often neglected the evaluation of domain-specific safety risks. To bridge this gap, we first define domain-specific safety principles for LLMs based on the AMA Principles of Medical Ethics, the ABA Model Rules of Professional Conduct, and the CFA Institute Code of Ethics. Building on this foundation, we introduce Trident-Bench, a benchmark specifically targeting LLM safety in the legal, financial, and medical domains. We evaluated 19 general-purpose and domain-specialized models on Trident-Bench and show that it effectively reveals key safety gaps -- strong generalist models (e.g., GPT, Gemini) can meet basic expectations, whereas domain-specialized models often struggle with subtle ethical nuances. This highlights an urgent need for finer-grained domain-specific safety improvements. By introducing Trident-Bench, our work provides one of the first systematic resources for studying LLM safety in law and finance, and lays the groundwork for future research aimed at reducing the safety risks of deploying LLMs in professionally regulated fields. Code and benchmark will be released at: https://github.com/zackhuiiiii/TRIDENT.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 53 of its 75 references that are in the map. First posted on arXiv 22 Jul 2025.

Builds on

53 of 75 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ToxiCraft: A Novel Framework for Synthetic Generation of Harmful Information23 Sep
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2024Mixtral of Expertscited by 1,785
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
38 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  8. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  9. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  10. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  11. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  12. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  13. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  14. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  15. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  16. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  17. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  18. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  19. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  20. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  21. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  22. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  23. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  24. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
  25. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  26. 2024Can LLM be a Personalized Judge?cited by 99
  27. 2024MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notescited by 77
  28. 2023FinGPT: Democratizing Internet-scale Data for Financial Large Language Modelscited by 65
  29. 2025International AI Safety Reportcited by 64
  30. 2024The Ethics of ChatGPT in Medicine and Healthcare: A Systematic Review on Large Language Models (LLMs)cited by 45
  31. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 43
  32. 2024LawLLM: Law Large Language Model for the US Legal Systemcited by 37
  33. 2023BizBench: A Quantitative Reasoning Benchmark for Business and Financecited by 28
  34. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  35. 2024Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilitiescited by 16
  36. 2024PropaInsight: Toward Deeper Understanding of Propaganda in Terms of Techniques, Appeals, and Intentcited by 9
  37. 2025Toward Safe and Human-Aligned Game Conversational Recommendation via Multi-Agent Decompositioncited by 9
  38. 2025The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMscited by 7

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026From RAG to Agentic RAG for Faithful Islamic Question Answeringcited by 9
  2. 2026LLM Agents in Law: Taxonomy, Applications, and Challengescited by 5
  3. 2026COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMscited by 1
  4. 2026What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Modelscited by 1
  5. 2025InvThink: Premortem Reasoning for Safer Language Modelscited by 0
  6. 2026PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Modelscited by 0
  7. 2026Lower-Resource, Higher Scores: Language Bias in LLM Evaluatorscited by 0