mlsanityData to 7 Oct 2026

Understanding and Mitigating Risks of Generative AI in Financial Services

arXiv 2504.20086 ↗First posted 25 Apr 2025cs.CL cs.AI cs.CYcited by 4

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

To responsibly develop Generative AI (GenAI) products, it is critical to define the scope of acceptable inputs and outputs. What constitutes a "safe" response is an actively debated question. Academic work puts an outsized focus on evaluating models by themselves for general purpose aspects such as toxicity, bias, and fairness, especially in conversational applications being used by a broad audience. In contrast, less focus is put on considering sociotechnical systems in specialized domains. Yet, those specialized systems can be subject to extensive and well-understood legal and regulatory scrutiny. These product-specific considerations need to be set in industry-specific laws, regulations, and corporate governance requirements. In this paper, we aim to highlight AI content safety considerations specific to the financial services domain and outline an associated AI content risk taxonomy. We compare this taxonomy to existing work in this space and discuss implications of risk category violations on various stakeholders. We evaluate how existing open-source technical guardrail solutions cover this taxonomy by assessing them on data collected via red-teaming activities. Our results demonstrate that these guardrails fail to detect most of the content risks we discuss.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 37 of its 97 references that are in the map. First posted on arXiv 25 Apr 2025.

Builds on

37 of 97 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Lessons From Red Teaming 100 Generative AI Products13 Jan
  2. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)20 Jul
  3. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  4. 2023From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy3 Jul
  5. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch15 Jun
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 30

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  7. 2021Ethical and social risks of harm from Language Modelscited by 957
  8. 2023BloombergGPT: A Large Language Model for Financecited by 923
  9. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  10. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  11. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  12. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
18 more
  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023FinGPT: Open-Source Financial Large Language Modelscited by 244
  3. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  4. 2023Large Language Models in Finance: A Surveycited by 223
  5. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  6. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  7. 2023Model evaluation for extreme riskscited by 167
  8. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  9. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  10. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  11. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  12. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  13. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  14. 2022Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governancecited by 67
  15. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  16. 2022A Hazard Analysis Framework for Code Synthesis Large Language Modelscited by 22
  17. 2024STAND-Guard: A Small Task-Adaptive Content Moderation Modelcited by 11
  18. 2024Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitationscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domaincited by 6
  2. 2026FinVault: Benchmarking Financial Agent Safety in Execution-Grounded Environmentscited by 6
  3. 2026StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generationcited by 4
  4. 2025Reward Models are Metrics in a Trench Coatcited by 2