# Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment

arXiv: https://arxiv.org/abs/2202.02838 · HTML: https://mlsanity.com/paper/2202.02838/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 99 references that are in the map.
- First posted on arXiv 6 Feb 2022.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [The Frontiers of Fairness in Machine Learning](https://mlsanity.com/paper/1810.08810/) (2018)
- [Harnessing Deep Neural Networks with Logic Rules](https://mlsanity.com/paper/1603.06318/) (2016)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?](https://mlsanity.com/paper/1606.03556/) (2016)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)
- [FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learning](https://mlsanity.com/paper/1904.05419/) (2019)
- [Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS](https://mlsanity.com/paper/1912.06059/) (2019)
- [Aligning Faithful Interpretations with their Social Attribution](https://mlsanity.com/paper/2006.01067/) (2020)
- [Learning Optimal and Fair Decision Trees for Non-Discriminative Decision-Making](https://mlsanity.com/paper/1903.10598/) (2019)
- [Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs](https://mlsanity.com/paper/2004.11440/) (2020)
- [Incorporating Priors with Feature Attribution on Text Classification](https://mlsanity.com/paper/1906.08286/) (2019)
- [RuleMatrix: Visualizing and Understanding Classifiers with Rules](https://mlsanity.com/paper/1807.06228/) (2018)
- [Do Convolutional Neural Networks Learn Class Hierarchy?](https://mlsanity.com/paper/1710.06501/) (2017)
- [ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models](https://mlsanity.com/paper/1704.01942/) (2017)

## Built on by (most cited first, 27 of 27)

- [Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning](https://mlsanity.com/paper/2212.03954/) (2022)
- [GazeGNN: A Gaze-Guided Graph Neural Network for Chest X-ray Classification](https://mlsanity.com/paper/2305.18221/) (2023)
- [RES: A Robust Framework for Guiding Visual Explanation](https://mlsanity.com/paper/2206.13413/) (2022)
- [Studying How to Efficiently and Effectively Guide Models with Explanations](https://mlsanity.com/paper/2303.11932/) (2023)
- [Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?](https://mlsanity.com/paper/2306.01220/) (2023)
- [Good Teachers Explain: Explanation-Enhanced Knowledge Distillation](https://mlsanity.com/paper/2402.03119/) (2024)
- [VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives](https://mlsanity.com/paper/2206.11212/) (2022)
- [Saliency-Bench: A Comprehensive Benchmark for Evaluating Visual Explanations](https://mlsanity.com/paper/2310.08537/) (2023)
- [SLIM: Spuriousness Mitigation with Minimal Human Annotations](https://mlsanity.com/paper/2407.05594/) (2024)
- [Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproducts](https://mlsanity.com/paper/2303.17595/) (2023)
- [DUE: Dynamic Uncertainty-Aware Explanation Supervision via 3D Imputation](https://mlsanity.com/paper/2403.10831/) (2024)
- [Designing a Direct Feedback Loop between Humans and Convolutional Neural Networks through Local Explanations](https://mlsanity.com/paper/2307.04036/) (2023)
- [On the Interplay of Human-AI Alignment,Fairness, and Performance Trade-offs in Medical Imaging](https://mlsanity.com/paper/2505.10231/) (2025)
- [Visual Attention Prompted Prediction and Learning](https://mlsanity.com/paper/2310.08420/) (2023)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)
- [FastRM: An efficient and automatic explainability framework for multimodal generative models](https://mlsanity.com/paper/2412.01487/) (2024)
- [AIM: Amending Inherent Interpretability via Self-Supervised Masking](https://mlsanity.com/paper/2508.11502/) (2025)
- [ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement](https://mlsanity.com/paper/2509.13282/) (2025)
- [Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals](https://mlsanity.com/paper/2602.08882/) (2026)
- [Following the Diagnostic Trace: Visual Cognition-guided Cooperative Network for Chest X-Ray Diagnosis](https://mlsanity.com/paper/2602.21657/) (2026)
- [Explanation-based Training with Differentiable Insertion/Deletion Metric-aware Regularizers](https://mlsanity.com/paper/2310.12553/) (2023)
- [3DPFIX: Improving Remote Novices' 3D Printing Troubleshooting through Human-AI Collaboration](https://mlsanity.com/paper/2401.15877/) (2024)
- [Effective Guidance for Model Attention with Simple Yes-no Annotations](https://mlsanity.com/paper/2410.22312/) (2024)
- [From Attribution to Action: Jointly ALIGNing Predictions and Explanations](https://mlsanity.com/paper/2511.06944/) (2025)
- [Hybrid Attribution Priors for Explainable and Robust Model Training](https://mlsanity.com/paper/2512.14719/) (2025)
- [Explanatory Interactive Machine Learning for Bias Mitigation in Visual Gender Classification](https://mlsanity.com/paper/2602.13286/) (2026)
- [Steering Neural Network Training through Interpretable Constraints Based on Partial Dependence](https://mlsanity.com/paper/2607.08641/) (2026)

## Abstract (arXiv)

While Deep Neural Networks (DNNs) are deriving the major innovations in nearly every field through their powerful automation, we are also witnessing the peril behind automation as a form of bias, such as automated racism, gender bias, and adversarial bias. As the societal impact of DNNs grows, finding an effective way to steer DNNs to align their behavior with the human mental model has become indispensable in realizing fair and accountable models. We propose a novel framework of Interactive Attention Alignment (IAA) that aims at realizing human-steerable Deep Neural Networks (DNNs). IAA leverages DNN model explanation method as an interactive medium that humans can use to unveil the cases of biased model attention and directly adjust the attention. In improving the DNN using human-generated adjusted attention, we introduce GRADIA, a novel computational pipeline that jointly maximizes attention quality and prediction accuracy. We evaluated IAA framework in Study 1 and GRADIA in Study 2 in a gender classification problem. Study 1 found applying IAA can significantly improve the perceived quality of model attention from human eyes. In Study 2, we found using GRADIA can (1) significantly improve the perceived quality of model attention and (2) significantly improve model performance in scenarios where the training samples are limited. We present implications for future interactive user interfaces design towards human-alignable AI.