# SAIF: A Comprehensive Framework for Evaluating the Risks of Generative AI in the Public Sector

arXiv: https://arxiv.org/abs/2501.08814 · HTML: https://mlsanity.com/paper/2501.08814/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 15 Jan 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 19 of 19)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://mlsanity.com/paper/2311.03191/) (2023)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Generating Images with Multimodal Language Models](https://mlsanity.com/paper/2305.17216/) (2023)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models](https://mlsanity.com/paper/2402.13457/) (2024)
- [Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts](https://mlsanity.com/paper/2311.09127/) (2023)
- [InGram: Inductive Knowledge Graph Embedding via Relation Graphs](https://mlsanity.com/paper/2305.19987/) (2023)
- [AI Risk Categorization Decoded (AIR 2024): From Government Regulations to Corporate Policies](https://mlsanity.com/paper/2406.17864/) (2024)
- [Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation](https://mlsanity.com/paper/2406.11852/) (2024)
- [Representation Learning on Hyper-Relational and Numeric Knowledge Graphs with Transformers](https://mlsanity.com/paper/2305.18256/) (2023)
- [Generative AI is already widespread in the public sector](https://mlsanity.com/paper/2401.01291/) (2024)
- [Learning Representations of Bi-level Knowledge Graphs for Reasoning beyond Link Prediction](https://mlsanity.com/paper/2302.02601/) (2023)
- [Exfiltration of personal information from ChatGPT via prompt injection](https://mlsanity.com/paper/2406.00199/) (2024)
- [U Can't Gen This? A Survey of Intellectual Property Protection Methods for Data in Generative AI](https://mlsanity.com/paper/2406.15386/) (2024)
- [Applications of Generative AI in Healthcare: algorithmic, ethical, legal and societal considerations](https://mlsanity.com/paper/2406.10632/) (2024)
- [Toward Safe Evolution of Artificial Intelligence (AI) based Conversational Agents to Support Adolescent Mental and Sexual Health Knowledge Discovery](https://mlsanity.com/paper/2404.03023/) (2024)

## Built on by (most cited first, 2 of 2)

- [Sovereign AI-based Public Services are Viable and Affordable](https://mlsanity.com/paper/2603.01869/) (2026)
- [AI Deployment and Cyber Governance Failures in Public-Sector Organizations: A Typological Analysis](https://mlsanity.com/paper/2607.25368/) (2026)

## Abstract (arXiv)

The rapid adoption of generative AI in the public sector, encompassing diverse applications ranging from automated public assistance to welfare services and immigration processes, highlights its transformative potential while underscoring the pressing need for thorough risk assessments. Despite its growing presence, evaluations of risks associated with AI-driven systems in the public sector remain insufficiently explored. Building upon an established taxonomy of AI risks derived from diverse government policies and corporate guidelines, we investigate the critical risks posed by generative AI in the public sector while extending the scope to account for its multimodal capabilities. In addition, we propose a Systematic dAta generatIon Framework for evaluating the risks of generative AI (SAIF). SAIF involves four key stages: breaking down risks, designing scenarios, applying jailbreak methods, and exploring prompt types. It ensures the systematic and consistent generation of prompt data, facilitating a comprehensive evaluation while providing a solid foundation for mitigating the risks. Furthermore, SAIF is designed to accommodate emerging jailbreak methods and evolving prompt types, thereby enabling effective responses to unforeseen risk scenarios. We believe that this study can play a crucial role in fostering the safe and responsible integration of generative AI into the public sector.