mlsanityData to 14 Oct 2026

LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images

arXiv 2305.19164 ↗First posted 30 May 2023cs.CV cs.CL cs.LGcited by 46

Not on the map. Why?

Abstract (arXiv)

We propose an automated algorithm to stress-test a trained visual model by generating language-guided counterfactual test images (LANCE). Our method leverages recent progress in large language modeling and text-based image editing to augment an IID test set with a suite of diverse, realistic, and challenging test images without altering model weights. We benchmark the performance of a diverse set of pre-trained models on our generated data and observe significant and consistent performance drops. We further analyze model sensitivity across different types of edits, and demonstrate its applicability at surfacing previously unknown class-level model biases in ImageNet. Code is available at https://github.com/virajprabhu/lance.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 62 of its 74 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

62 of 74 references.

References · 62

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2022Training language models to follow instructions with human feedbackcited by 15,971
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  11. 2020Denoising Diffusion Implicit Modelscited by 9,781
  12. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
38 more
  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2022Generative Adversarial Networkscited by 5,627
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2022A ConvNet for the 2020scited by 3,579
  13. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  14. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  15. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  16. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  17. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  18. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  19. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  20. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  21. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  22. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  23. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  24. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  25. 2015A Neural Algorithm of Artistic Stylecited by 483
  26. 2017Semantic Foggy Scene Understanding with Synthetic Datacited by 403
  27. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  28. 2019Counterfactual Visual Explanationscited by 313
  29. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  30. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 156
  31. 2023Delta Denoising Scorecited by 121
  32. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  33. 2021Counterfactual Generative Networkscited by 111
  34. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 108
  35. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  36. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  37. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  38. 2022Diffusion Models for Counterfactual Explanationscited by 79

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 21; 2025: 15; 2026: 4.

46 papers.

Sort:

By area:

  1. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 77
  2. 2024Probing Visual Language Priors in VLMscited by 45
  3. 2024A Sober Look at the Robustness of CLIPs to Spurious Featurescited by 38
  4. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMscited by 33
  5. 2023RadEdit: stress-testing biomedical vision models via diffusion image editingcited by 28
  6. 2023Reliability in Semantic Segmentation: Can We Use Synthetic Data?cited by 24
  7. 2024ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Objectcited by 22
  8. 2024Evolving Interpretable Visual Classifiers with Large Language Modelscited by 18
  9. 2024Semi-Truths: A Large-Scale Dataset of AI-Augmented Images for Evaluating Robustness of AI-Generated Image detectorscited by 17
  10. 2025PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusioncited by 17
  11. 2024Global Counterfactual Directionscited by 15
  12. 2024Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Modelscited by 11
34 more
  1. 2025Diffusion Counterfactual Generation with Semantic Abductioncited by 10
  2. 2023GeNIe: Generative Hard Negative Images Through Diffusioncited by 8
  3. 2024Contrast Sets for Evaluating Language-Guided Robot Policiescited by 7
  4. 2024Benchmarking Segmentation Models with Mask-Preserved Attribute Editingcited by 6
  5. 2024Visual Data Diagnosis and Debiasing with Concept Graphscited by 6
  6. 2025Causally Steered Diffusion for Automated Video Counterfactual Generationcited by 6
  7. 2023ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlationscited by 5
  8. 2024ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changescited by 5
  9. 2025Multiverse Through Deepfakes: The MultiFakeVerse Dataset of Person-Centric Visual and Conceptual Manipulationscited by 5
  10. 2024Unsupervised Model Diagnosiscited by 4
  11. 2024Evaluating Model Bias Requires Characterizing its Mistakescited by 3
  12. 2024Finding Patterns in Ambiguity: Interpretable Stress Testing in the Decision~Boundarycited by 2
  13. 2024GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifierscited by 2
  14. 2025Behind the Screens: Uncovering Bias in AI-Driven Video Interview Assessments Using Counterfactualscited by 2
  15. 2025CNS-Bench: Benchmarking Image Classifier Robustness Under Continuous Nuisance Shiftscited by 2
  16. 2025Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generationcited by 2
  17. 2025SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generationcited by 2
  18. 2026Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editingcited by 2
  19. 2023Evaluation of human-model prediction difference on the Internet Scale of Datacited by 1
  20. 2024Reinforcing Pre-trained Models Using Counterfactual Imagescited by 1
  21. 2024DEPICT: Diffusion-Enabled Permutation Importance for Image Classification Taskscited by 1
  22. 2024Are Pose Estimators Ready for the Open World? STAGE: A GenAI Toolkit for Auditing 3D Human Pose Estimatorscited by 1
  23. 2025Leveraging Vision-Language Foundation Models to Reveal Hidden Image-Attribute Relationships in Medical Imagingcited by 1
  24. 2025RO-Bench: Large-scale robustness evaluation of MLLMs with text-driven counterfactual videoscited by 1
  25. 2026Personalized Generative Models for Contextual Debiasingcited by 1
  26. 2024Linking in Style: Understanding learned features in deep learning modelscited by 0
  27. 2024LLM-Assisted Red Teaming of Diffusion Models through "Failures Are Fated, But Can Be Faded"cited by 0
  28. 2025Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generationcited by 0
  29. 2025Learning Counterfactual Outcomes Under Rank Preservationcited by 0
  30. 2025Towards Assessing Deep Learning Test Input Generatorscited by 0
  31. 2025Blind Spot Navigation: Evolutionary Discovery of Sensitive Semantic Concepts for LVLMscited by 0
  32. 2025Cross-modal Counterfactual Explanations: Uncovering Decision Factors and Dataset Biases in Subjective Classificationcited by 0
  33. 2026SeFaR: Semantic Feature-aware Robustness Testing of Deep Neural Networkscited by 0
  34. 2026From Pixels to Policy: A Multi-Agent System for Intervention and Geo-Spatial Decision Supportcited by 0