# Characteristics and prevalence of fake social media profiles with AI-generated faces

arXiv: https://arxiv.org/abs/2401.02627 · HTML: https://mlsanity.com/paper/2401.02627/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 57 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Builds on (most cited first, 30 of 30)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models](https://mlsanity.com/paper/2303.10420/) (2023)
- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations](https://mlsanity.com/paper/2301.04246/) (2023)
- [Detecting GAN generated Fake Images using Co-occurrence Matrices](https://mlsanity.com/paper/1903.06836/) (2019)
- [Towards the Detection of Diffusion Model Deepfakes](https://mlsanity.com/paper/2210.14571/) (2022)
- [Human heuristics for AI-generated language are flawed](https://mlsanity.com/paper/2206.07271/) (2022)
- [The Rise of Social Bots](https://mlsanity.com/paper/1407.5225/) (2014)
- [Scalable and Generalizable Social Bot Detection through Data Selection](https://mlsanity.com/paper/1911.09179/) (2019)
- [Exposing GAN-generated Faces Using Inconsistent Corneal Specular Highlights](https://mlsanity.com/paper/2009.11924/) (2020)
- [Uncovering Coordinated Networks on Social Media: Methods and Case Studies](https://mlsanity.com/paper/2001.05658/) (2020)
- [Eyes Tell All: Irregular Pupil Shapes Reveal GAN-generated Faces](https://mlsanity.com/paper/2109.00162/) (2021)
- [Anatomy of an AI-powered malicious social botnet](https://mlsanity.com/paper/2307.16336/) (2023)
- [Detection of Novel Social Bots by Ensembles of Specialized Classifiers](https://mlsanity.com/paper/2006.06867/) (2020)
- [Exposing GAN-synthesized Faces Using Landmark Locations](https://mlsanity.com/paper/1904.00167/) (2019)
- [GAN-generated Faces Detection: A Survey and New Perspectives](https://mlsanity.com/paper/2202.07145/) (2022)
- [Arming the public with artificial intelligence to counter social bots](https://mlsanity.com/paper/1901.00912/) (2019)
- [More Real than Real: A Study on Human Visual Perception of Synthetic Faces](https://mlsanity.com/paper/2106.07226/) (2021)
- [Testing Human Ability To Detect Deepfake Images of Human Faces](https://mlsanity.com/paper/2212.05056/) (2022)
- [Just Another Day on Twitter: A Complete 24 Hours of Twitter Data](https://mlsanity.com/paper/2301.11429/) (2023)
- [The Value of AI Guidance in Human Examination of Synthetically-Generated Faces](https://mlsanity.com/paper/2208.10544/) (2022)

## Built on by (most cited first, 14 of 14)

- [How malicious AI swarms can threaten democracy: The fusion of agentic AI and LLMs marks a new frontier in information warfare](https://mlsanity.com/paper/2506.06299/) (2025)
- [GAMED: Knowledge Adaptive Multi-Experts Decoupling for Multimodal Fake News Detection](https://mlsanity.com/paper/2412.12164/) (2024)
- [AI-Generated Faces in the Real World: A Large-Scale Case Study of Twitter Profile Images](https://mlsanity.com/paper/2404.14244/) (2024)
- [TrueFake: A Real World Case Dataset of Last Generation Fake Images also Shared on Social Networks](https://mlsanity.com/paper/2504.20658/) (2025)
- [RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors](https://mlsanity.com/paper/2506.03988/) (2025)
- [Synthetic Photography Detection: A Visual Guidance for Identifying Synthetic Images Created by AI](https://mlsanity.com/paper/2408.06398/) (2024)
- [Graph-based Fake Account Detection: A Survey](https://mlsanity.com/paper/2507.06541/) (2025)
- [A Marketplace for AI-Generated Adult Content and Deepfakes](https://mlsanity.com/paper/2601.09117/) (2026)
- [Prefill-Guided Thinking for zero-shot detection of AI-generated images](https://mlsanity.com/paper/2506.11031/) (2025)
- [Identifying Models Behind Text-to-Image Leaderboards](https://mlsanity.com/paper/2601.09647/) (2026)
- [Efficient Zero-Shot AI-Generated Image Detection](https://mlsanity.com/paper/2603.21619/) (2026)
- [Collective Opinion Dynamics in Structured LLM Populations](https://mlsanity.com/paper/2604.11312/) (2026)
- [Adversarial Creation and Detection of AI-Generated Social Bot Content](https://mlsanity.com/paper/2606.07219/) (2026)
- [Frontier vision-language models have overtaken young adults at detecting AI-generated portraits -- but not their calibration](https://mlsanity.com/paper/2608.30210/) (2026)

## Abstract (arXiv)

Recent advancements in generative artificial intelligence (AI) have raised concerns about their potential to create convincing fake social media accounts, but empirical evidence is lacking. In this paper, we present a systematic analysis of Twitter (X) accounts using human faces generated by Generative Adversarial Networks (GANs) for their profile pictures. We present a dataset of 1,420 such accounts and show that they are used to spread scams, spam, and amplify coordinated messages, among other inauthentic activities. Leveraging a feature of GAN-generated faces -- consistent eye placement -- and supplementing it with human annotation, we devise an effective method for identifying GAN-generated profiles in the wild. Applying this method to a random sample of active Twitter users, we estimate a lower bound for the prevalence of profiles using GAN-generated faces between 0.021% and 0.044% -- around 10K daily active accounts. These findings underscore the emerging threats posed by multimodal generative AI. We release the source code of our detection method and the data we collect to facilitate further investigation. Additionally, we provide practical heuristics to assist social media users in recognizing such accounts.