# Exploring the Use of Abusive Generative AI Models on Civitai

arXiv: https://arxiv.org/abs/2407.12876 · HTML: https://mlsanity.com/paper/2407.12876/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 12 of 12)

- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [Art and the science of generative AI: A deeper dive](https://mlsanity.com/paper/2306.04141/) (2023)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [Understanding and Creating Art with AI: Review and Outlook](https://mlsanity.com/paper/2102.09109/) (2021)
- [Typology of Risks of Generative Text-to-Image Models](https://mlsanity.com/paper/2307.05543/) (2023)
- [Copyright in Generative Deep Learning](https://mlsanity.com/paper/2105.09266/) (2021)
- [Deepfake Videos in the Wild: Analysis and Detection](https://mlsanity.com/paper/2103.04263/) (2021)
- [Are Deepfakes Concerning? Analyzing Conversations of Deepfakes on Reddit and Exploring Societal Implications](https://mlsanity.com/paper/2203.15044/) (2022)
- [Identifying and Characterizing Active Citizens who Refute Misinformation in Social Media](https://mlsanity.com/paper/2204.10080/) (2022)
- [Interplay between Social Influence and Network Centrality: A Comparative Study on Shapley Centrality and Single-Node-Influence Centrality](https://mlsanity.com/paper/1602.03780/) (2016)
- [A Study of Partisan News Sharing in the Russian invasion of Ukraine](https://mlsanity.com/paper/2311.15294/) (2023)
- [torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP](https://mlsanity.com/paper/2310.17644/) (2023)

## Built on by (most cited first, 10 of 10)

- [Characterizing LLM-driven Social Network: The Chirper.ai Case](https://mlsanity.com/paper/2504.10286/) (2025)
- [Unveiling AI's Threats to Child Protection: Regulatory efforts to Criminalize AI-Generated CSAM and Emerging Children's Rights Violations](https://mlsanity.com/paper/2503.00433/) (2025)
- ["Unlimited Realm of Exploration and Experimentation": Methods and Motivations of AI-Generated Sexual Content Creators](https://mlsanity.com/paper/2601.21028/) (2026)
- [Delta Sampling: Data-Free Knowledge Transfer Across Diffusion Models](https://mlsanity.com/paper/2512.03056/) (2025)
- [LoRAShield: Data-Free Editing Alignment for Secure Personalized LoRA Sharing](https://mlsanity.com/paper/2507.07056/) (2025)
- [A Marketplace for AI-Generated Adult Content and Deepfakes](https://mlsanity.com/paper/2601.09117/) (2026)
- [Gradients: When Markets Meet Fine-tuning -- A Distributed Approach to Model Optimisation](https://mlsanity.com/paper/2506.07940/) (2025)
- [CARLoS: Retrieval via Concise Assessment Representation of LoRAs at Scale](https://mlsanity.com/paper/2512.08826/) (2025)
- [Synthetic Sociality: How Generative Models Privatize the Social Fabric](https://mlsanity.com/paper/2605.14090/) (2026)
- [Harmful Content Generation in Text-to-Image Models: Capabilities and Moderation Limitations](https://mlsanity.com/paper/2610.06503/) (2026)

## Abstract (arXiv)

The rise of generative AI is transforming the landscape of digital imagery, and exerting a significant influence on online creative communities. This has led to the emergence of AI-Generated Content (AIGC) social platforms, such as Civitai. These distinctive social platforms allow users to build and share their own generative AI models, thereby enhancing the potential for more diverse artistic expression. Designed in the vein of social networks, they also provide artists with the means to showcase their creations (generated from the models), engage in discussions, and obtain feedback, thus nurturing a sense of community. Yet, this openness also raises concerns about the abuse of such platforms, e.g., using models to disseminate deceptive deepfakes or infringe upon copyrights. To explore this, we conduct the first comprehensive empirical study of an AIGC social platform, focusing on its use for generating abusive content. As an exemplar, we construct a comprehensive dataset covering Civitai, the largest available AIGC social platform. Based on this dataset of 87K models and 2M images, we explore the characteristics of content and discuss strategies for moderation to better govern these platforms.