mlsanityData to 7 Oct 2026

Gen-AI for User Safety: A Survey

arXiv 2411.06606 ↗First posted 10 Nov 2024cs.AI cs.CRcited by 2

Not on the map. Why?

Abstract (arXiv)

Machine Learning and data mining techniques (i.e. supervised and unsupervised techniques) are used across domains to detect user safety violations. Examples include classifiers used to detect whether an email is spam or a web-page is requesting bank login information. However, existing ML/DM classifiers are limited in their ability to understand natural languages w.r.t the context and nuances. The aforementioned challenges are overcome with the arrival of Gen-AI techniques, along with their inherent ability w.r.t translation between languages, fine-tuning between various tasks and domains. In this manuscript, we provide a comprehensive overview of the various work done while using Gen-AI techniques w.r.t user safety. In particular, we first provide the various domains (e.g. phishing, malware, content moderation, counterfeit, physical safety) across which Gen-AI techniques have been applied. Next, we provide how Gen-AI techniques can be used in conjunction with various data modalities i.e. text, images, videos, audio, executable binaries to detect violations of user-safety. Further, also provide an overview of how Gen-AI techniques can be used in an adversarial setting. We believe that this work represents the first summarization of Gen-AI techniques for user-safety.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 56 of its 110 references that are in the map. First posted on arXiv 10 Nov 2024.

Builds on

56 of 110 references.

References · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2022A Survey on In-context Learningcited by 717
  7. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  8. 2023Towards Universal Fake Image Detectors that Generalize Across Generative Modelscited by 483
  9. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  10. 2021FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Datasetcited by 179
  11. 2023Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Methodcited by 97
  12. 2021WaveFake: A Data Set to Facilitate Audio Deepfake Detectioncited by 78
38 more
  1. 2023Fake News in Sheep's Clothing: Robust Fake News Detection Against LLM-Empowered Style Attackscited by 69
  2. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  3. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Modelscited by 57
  4. 2024MLAAD: The Multi-Language Audio Anti-Spoofing Datasetcited by 55
  5. 2024UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Imagescited by 52
  6. 2023Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devicescited by 38
  7. 2024Evidence-Driven Retrieval Augmented Response Generation for Online Misinformationcited by 32
  8. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  9. 2019A Transformer-based approach to Irony and Sarcasm detectioncited by 28
  10. 2024Pushing The Limit of LLM Capacity for Text Classificationcited by 28
  11. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you thinkcited by 27
  12. 2023Digital Deception: Generative Artificial Intelligence in Social Engineering and Phishingcited by 26
  13. 2024ChatSpamDetector: Leveraging Large Language Models for Effective Phishing Email Detectioncited by 25
  14. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  15. 2024An Investigation of Large Language Models for Real-World Hate Speech Detectioncited by 20
  16. 2024LLM-Assisted Crisis Management: Building Advanced LLM Platforms for Effective Emergency Response and Public Collaborationcited by 20
  17. 2024Assessing LLMs in Malicious Code Deobfuscation of Real-world Malware Campaignscited by 18
  18. 2023RatGPT: Turning online LLMs into Proxies for Malware Attackscited by 17
  19. 2024AppPoet: Large Language Model based Android malware detection via multi-view prompt engineeringcited by 16
  20. 2024PolyGlotFake: A Novel Multilingual and Multimodal DeepFake Datasetcited by 16
  21. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 16
  22. 2023Protecting Society from AI Misuse: When are Restrictions on Capabilities Warranted?cited by 15
  23. 2024Defending Against Social Engineering Attacks in the Age of LLMscited by 15
  24. 2023Adapting Fake News Detection to the Era of Large Language Modelscited by 14
  25. 2023Fine-Tuning Llama 2 Large Language Models for Detecting Online Sexual Predatory Chats and Abusive Textscited by 13
  26. 2024Moderating Illicit Online Image Promotion for Unsafe User-Generated Content Games Using Large Vision-Language Modelscited by 13
  27. 2024Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossingcited by 12
  28. 2024Scaling Up LLM Reviews for Google Ads Content Moderationcited by 12
  29. 2024Towards Understanding Unsafe Video Generationcited by 10
  30. 2023Detecting Phishing Sites Using ChatGPTcited by 9
  31. 2024A Survey of Malware Detection Using Deep Learningcited by 8
  32. 2024Vision Language Model for Interpretable and Fine-grained Detection of Safety Compliance in Diverse Workplacescited by 8
  33. 2024A Comprehensive Study on NLP Data Augmentation for Hate Speech Detection: Legacy Methods, BERT, and LLMscited by 7
  34. 2024LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languagescited by 7
  35. 2024AI Safety Landscape for Large Language Models: Taxonomy, State-of-the-art, and Future Directionscited by 7
  36. 2024"I Like Sunnie More Than I Expected!": Exploring User Expectation and Perception of an Anthropomorphic LLM-based Conversational Agent for Well-Being Supportcited by 5
  37. 2024DetoxBench: Benchmarking Large Language Models for Multitask Fraud & Abuse Detectioncited by 5
  38. 2024Code-mixed Sentiment and Hate-speech Predictioncited by 4

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Modelscited by 29
  2. 2025Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directionscited by 3