mlsanityData to 13 Oct 2026

FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios

arXiv 2605.00706 ↗First posted 1 May 2026cs.CLcited by 3

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Large language models (LLMs) are increasingly applied in financial scenarios. However, they may produce harmful outputs, including facilitating illegal activities or unethical behavior, posing serious compliance risks. To systematically evaluate LLM safety in finance, we propose FinSafetyBench, a bilingual (English-Chinese) red-teaming benchmark designed to test an LLM's refusal of requests that violate financial compliance. Grounded in real-world financial crime cases and ethics standards, the benchmark comprises 14 subcategories spanning financial crimes and ethical violations. Through extensive experiments on general-purpose and finance-specialized LLMs under three representative attack settings, we identify critical vulnerabilities that allow adversarial prompts to bypass compliance safeguards. Further analysis reveals stronger susceptibility in Chinese contexts and highlights the limitations of prompt-level defenses against sophisticated or implicit manipulation strategies.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 41 of its 53 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

41 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Flames: Benchmarking Value Alignment of LLMs in Chinese12 Nov
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 39

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,064
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,405
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,404
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,112
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 664
  7. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 565
  8. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 553
  9. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 429
  10. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 235
  11. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 222
  12. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
27 more
  1. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 196
  2. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  3. 2023XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameterscited by 92
  4. 2025AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agentscited by 86
  5. 2023InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuningcited by 81
  6. 2024INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agentcited by 63
  7. 2023BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmarkcited by 60
  8. 2024Large Language Model Agent in Financial Trading: A Surveycited by 60
  9. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  10. 2024Distract Large Language Models for Automatic Jailbreak Attackcited by 45
  11. 2025Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducementscited by 32
  12. 2025FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challengingcited by 28
  13. 2024Evaluating LLMs' Mathematical Reasoning in Financial Document Question Answeringcited by 25
  14. 2024Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Datasetcited by 23
  15. 2025BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMscited by 22
  16. 2023SC-Safety: A Multi-round Open-ended Question Adversarial Safety Benchmark for Large Language Models in Chinesecited by 17
  17. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 15
  18. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  19. 2024SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applicationscited by 11
  20. 2024FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Onlycited by 11
  21. 2024SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkagecited by 11
  22. 2025FinMaster: A Holistic Benchmark for Mastering Full-Pipeline Financial Workflows with LLMscited by 9
  23. 2024SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characterscited by 6
  24. 2024Modeling News Interactions and Influence for Financial Market Predictioncited by 6
  25. 2025SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilitiescited by 6
  26. 2024Compound-QA: A Benchmark for Evaluating LLMs on Compound Questionscited by 4
  27. 2025Program of Thoughts for Financial Reasoning: Leveraging Dynamic In-Context Examples and Generative Retrievalcited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languagescited by 3
  2. 2026When Financial Fine-tuning Fails: A Three-Level Detectability Analysis of Numerical Hallucination in Domain-Adapted Language Modelscited by 0
  3. 2026FinRT: Distilling Adaptive Red-Teaming Strategies into Reusable Adversarial Generators in Consumer Financecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming
  2. 2025Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domain
  3. 2026CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns
  4. 2024Chat Bankman-Fried: an Exploration of LLM Alignment in Finance
  5. 2024Developing Assurance Cases for Adversarial Robustness and Regulatory Compliance in LLMs
  6. 2026FinVault: Benchmarking Financial Agent Safety in Execution-Grounded Environments
  7. 2026Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
  8. 2026StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generation