# Addressing Weak Decision Boundaries in Image Classification by Leveraging Web Search and Generative Models

arXiv: https://arxiv.org/abs/2310.19986 · HTML: https://mlsanity.com/paper/2310.19986/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 63 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [Salient ImageNet: How to discover spurious features in Deep Learning?](https://mlsanity.com/paper/2110.04301/) (2021)
- [Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models](https://mlsanity.com/paper/2110.07736/) (2021)
- [Robustness to Spurious Correlations in Text Classification via Automatically Generated Counterfactuals](https://mlsanity.com/paper/2012.10040/) (2020)
- [Conscientious Classification: A Data Scientist's Guide to Discrimination-Aware Classification](https://mlsanity.com/paper/1907.09013/) (2019)
- [Exploring Racial Bias within Face Recognition via per-subject Adversarially-Enabled Data Augmentation](https://mlsanity.com/paper/2004.08945/) (2020)
- [Deflating Dataset Bias Using Synthetic Data Augmentation](https://mlsanity.com/paper/2004.13866/) (2020)
- [Finding and Fixing Spurious Patterns with Explanations](https://mlsanity.com/paper/2106.02112/) (2021)
- [Potential sources of dataset bias complicate investigation of underdiagnosis by machine learning algorithms](https://mlsanity.com/paper/2201.07856/) (2022)
- [Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curation](https://mlsanity.com/paper/2106.09467/) (2021)
- [Exploring Bias in GAN-based Data Augmentation for Small Samples](https://mlsanity.com/paper/1905.08495/) (2019)
- [Real2Sim or Sim2Real: Robotics Visual Insertion using Deep Reinforcement Learning and Real2Sim Policy Adaptation](https://mlsanity.com/paper/2206.02679/) (2022)
- [Robust Training Using Natural Transformation](https://mlsanity.com/paper/2105.04070/) (2021)

## Built on by (most cited first, 1 of 1)

- [Detecting Spurious Correlations via Robust Visual Concepts in Real and AI-Generated Image Classification](https://mlsanity.com/paper/2311.01655/) (2023)

## Abstract (arXiv)

Machine learning (ML) technologies are known to be riddled with ethical and operational problems, however, we are witnessing an increasing thrust by businesses to deploy them in sensitive applications. One major issue among many is that ML models do not perform equally well for underrepresented groups. This puts vulnerable populations in an even disadvantaged and unfavorable position. We propose an approach that leverages the power of web search and generative models to alleviate some of the shortcomings of discriminative models. We demonstrate our method on an image classification problem using ImageNet's People Subtree subset, and show that it is effective in enhancing robustness and mitigating bias in certain classes that represent vulnerable populations (e.g., female doctor of color). Our new method is able to (1) identify weak decision boundaries for such classes; (2) construct search queries for Google as well as text for generating images through DALL-E 2 and Stable Diffusion; and (3) show how these newly captured training samples could alleviate population bias issue. While still improving the model's overall performance considerably, we achieve a significant reduction (77.30%) in the model's gender accuracy disparity. In addition to these improvements, we observed a notable enhancement in the classifier's decision boundary, as it is characterized by fewer weakspots and an increased separation between classes. Although we showcase our method on vulnerable populations in this study, the proposed technique is extendable to a wide range of problems and domains.