mlsanityData to 7 Oct 2026

Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?

arXiv 2202.06675 ↗First posted 14 Feb 2022cs.AI cs.CV cs.CYcited by 150

Not on the map. Why?

Cited by 71 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large datasets underlying much of current machine learning raise serious issues concerning inappropriate content such as offensive, insulting, threatening, or might otherwise cause anxiety. This calls for increased dataset documentation, e.g., using datasheets. They, among other topics, encourage to reflect on the composition of the datasets. So far, this documentation, however, is done manually and therefore can be tedious and error-prone, especially for large image datasets. Here we ask the arguably "circular" question of whether a machine can help us reflect on inappropriate content, answering Question 16 in Datasheets. To this end, we propose to use the information stored in pre-trained transformer models to assist us in the documentation process. Specifically, prompt-tuning based on a dataset of socio-moral values steers CLIP to identify potentially inappropriate content, therefore reducing human labor. We then document the inappropriate images found using word clouds, based on captions generated using a vision-language model. The documentations of two popular, large-scale computer vision datasets -- ImageNet and OpenImages -- produced this way suggest that machines can indeed help dataset creators to answer Question 16 on inappropriate image content.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Builds on 38 of its 56 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

38 of 56 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2018Datasheets for Datasetscited by 1,371
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
26 more
  1. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  2. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  3. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  5. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  6. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  7. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  8. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  9. 2021A Survey of Transformerscited by 399
  10. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  11. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  12. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  13. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  14. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  15. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  16. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  17. 2015Semi-supervised Sequence Learningcited by 198
  18. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  19. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  20. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  21. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  22. 2020Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biasescited by 107
  23. 2019Exploring Hate Speech Detection in Multimodal Publicationscited by 103
  24. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  25. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
  26. 2020Fortifying Toxic Speech Detectors Against Veiled Toxicitycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 16; 2024: 37; 2025: 57; 2026: 35.

150 papers.

Sort:

By area:

  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  3. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 239
  4. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  5. 2023MMA-Diffusion: MultiModal Attack on Diffusion Modelscited by 184
  6. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 171
  7. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Promptscited by 161
  8. 2023Fair Diffusion: Instructing Text-to-Image Generation Models on Fairnesscited by 149
  9. 2023Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Eraserscited by 118
  10. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  11. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
  12. 2024Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Modelscited by 68
38 more
  1. 2024Latent Guard: a Safety Framework for Text-to-image Generationcited by 68
  2. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  3. 2024Erasing Undesirable Influence in Diffusion Modelscited by 52
  4. 2024UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Imagescited by 52
  5. 2024Scissorhands: Scrub Data Influence via Connection Sensitivity in Networkscited by 50
  6. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  7. 2024LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Modelscited by 49
  8. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  9. 2023Towards Safe Self-Distillation of Internet-Scale Text-to-Image Diffusion Modelscited by 41
  10. 2024Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matterscited by 38
  11. 2023SurrogatePrompt: Bypassing the Safety Filter of Text-to-Image Models via Substitutioncited by 32
  12. 2024ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Userscited by 32
  13. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  14. 2024Universal Prompt Optimizer for Safe Text-to-Image Generationcited by 27
  15. 2024TraSCE: Trajectory Steering for Concept Erasurecited by 25
  16. 2024Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Modelscited by 24
  17. 2025CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Modelscited by 24
  18. 2023Implicit Concept Removal of Diffusion Modelscited by 23
  19. 2024MLLM-as-a-Judge for Image Safety without Human Labelingcited by 23
  20. 2022Does CLIP Know My Face?cited by 19
  21. 2024DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimizationcited by 19
  22. 2024Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearningcited by 19
  23. 2025Training-Free Safe Denoisers for Safe Use of Diffusion Modelscited by 19
  24. 2025T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attackscited by 18
  25. 2024AlignGuard: Scalable Safety Alignment for Text-to-Image Generationcited by 17
  26. 2025Hyperbolic Safety-Aware Vision-Language Modelscited by 17
  27. 2023Mitigating Inappropriateness in Image Generation: Can there be Value in Reflecting the World's Ugliness?cited by 16
  28. 2025T2I-RiskyPrompt: A Benchmark for Safety Evaluation, Attack, and Defense on Text-to-Image Modelcited by 16
  29. 2024Unstable Unlearning: The Hidden Risk of Concept Resurgence in Diffusion Modelscited by 14
  30. 2025TRCE: Towards Reliable Malicious Concept Erasure in Text-to-Image Diffusion Modelscited by 14
  31. 2024Multimodal Pragmatic Jailbreak on Text-to-image Modelscited by 13
  32. 2025Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimizationcited by 12
  33. 2023Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Modelscited by 11
  34. 2025Concept Corrector: Erase concepts on the fly for text-to-image diffusion modelscited by 11
  35. 2025Revoking Amnesia: RL-based Trajectory Optimization to Resurrect Erased Concepts in Diffusion Modelscited by 11
  36. 2024T2Vs Meet VLMs: A Scalable Multimodal Dataset for Visual Harmfulness Recognitioncited by 10
  37. 2024SteerDiff: Steering towards Safe Text-to-Image Diffusion Modelscited by 10
  38. 2024AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion modelscited by 10

50 most cited of 150 All 150 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Neglected Risks: The Disturbing Reality of Children's Images in Datasets and the Urgent Call for Accountability
  2. 2024Clean Evaluations on Contaminated Visual Language Models
  3. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly
  4. 2026ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution
  5. 2024Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM
  6. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  7. 2025Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability
  8. 2026Multimodal Large Language Models as Image Classifiers