# SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model

arXiv: https://arxiv.org/abs/2402.18068 · HTML: https://mlsanity.com/paper/2402.18068/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 34 of its 40 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16381/) (2023)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation](https://mlsanity.com/paper/2307.06350/) (2023)
- [AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment](https://mlsanity.com/paper/2306.04717/) (2023)

## Built on by (most cited first, 15 of 15)

- [Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2405.14828/) (2024)
- [Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detection](https://mlsanity.com/paper/2506.00979/) (2025)
- [Whitened CLIP as a Likelihood Surrogate of Images and Captions](https://mlsanity.com/paper/2505.06934/) (2025)
- [Detecting Human Artifacts from Text-to-Image Models](https://mlsanity.com/paper/2411.13842/) (2024)
- [DiffDoctor: Diagnosing Image Diffusion Models Before Treating](https://mlsanity.com/paper/2501.12382/) (2025)
- [Agentic Retoucher for Text-To-Image Generation](https://mlsanity.com/paper/2601.02046/) (2026)
- [Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment](https://mlsanity.com/paper/2412.00306/) (2024)
- [Synthetic Photography Detection: A Visual Guidance for Identifying Synthetic Images Created by AI](https://mlsanity.com/paper/2408.06398/) (2024)
- [Unveiling Perceptual Artifacts: A Fine-Grained Benchmark for Interpretable AI-Generated Image Detection](https://mlsanity.com/paper/2601.19430/) (2026)
- [COBRA: COmBinatorial Retrieval Augmentation for Few-Shot Adaptation](https://mlsanity.com/paper/2412.17684/) (2024)
- [GeneVA: A Dataset of Human Annotations for Generative Text to Video Artifacts](https://mlsanity.com/paper/2509.08818/) (2025)
- [Veritas++: Value-aware On-Policy Distillation for Perception-Enhanced AIGI Detection](https://mlsanity.com/paper/2607.27113/) (2026)
- [GenShield: Unified Detection and Artifact Correction for AI-Generated Images](https://mlsanity.com/paper/2605.16122/) (2026)
- [Synthesis and Perceptual Scaling of High Resolution Naturalistic Images Using Stable Diffusion](https://mlsanity.com/paper/2410.13034/) (2024)
- [VLM- and LLM-Driven Multi-Agent System for PET Image Denoising](https://mlsanity.com/paper/2608.13791/) (2026)

## Abstract (arXiv)

In the rapidly evolving area of image synthesis, a serious challenge is the presence of complex artifacts that compromise perceptual realism of synthetic images. To alleviate artifacts and improve quality of synthetic images, we fine-tune Vision-Language Model (VLM) as artifact classifier to automatically identify and classify a wide range of artifacts and provide supervision for further optimizing generative models. Specifically, we develop a comprehensive artifact taxonomy and construct a dataset of synthetic images with artifact annotations for fine-tuning VLM, named SynArtifact-1K. The fine-tuned VLM exhibits superior ability of identifying artifacts and outperforms the baseline by 25.66%. To our knowledge, this is the first time such end-to-end artifact classification task and solution have been proposed. Finally, we leverage the output of VLM as feedback to refine the generative model for alleviating artifacts. Visualization results and user study demonstrate that the quality of images synthesized by the refined diffusion model has been obviously improved.