# Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification

arXiv: https://arxiv.org/abs/2108.08728 · HTML: https://mlsanity.com/paper/2108.08728/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 70 references that are in the map.
- First posted on arXiv 19 Aug 2021.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline)](https://mlsanity.com/paper/1711.09349/) (2017)
- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://mlsanity.com/paper/2002.10857/) (2020)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Omni-Scale Feature Learning for Person Re-Identification](https://mlsanity.com/paper/1905.00953/) (2019)
- [Learning Discriminative Features with Multiple Granularities for Person Re-Identification](https://mlsanity.com/paper/1804.01438/) (2018)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Joint Discriminative and Generative Learning for Person Re-identification](https://mlsanity.com/paper/1904.07223/) (2019)
- [Harmonious Attention Network for Person Re-Identification](https://mlsanity.com/paper/1802.08122/) (2018)
- [CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training](https://mlsanity.com/paper/1709.02023/) (2017)
- [Visual Commonsense R-CNN](https://mlsanity.com/paper/2002.12204/) (2020)
- [Learning to Navigate for Fine-grained Classification](https://mlsanity.com/paper/1809.00287/) (2018)
- [Part-Aligned Bilinear Representations for Person Re-identification](https://mlsanity.com/paper/1804.07094/) (2018)
- [Learning Attentive Pairwise Interaction for Fine-Grained Classification](https://mlsanity.com/paper/2002.10191/) (2020)
- [Interaction-and-Aggregation Network for Person Re-identification](https://mlsanity.com/paper/1907.08435/) (2019)
- [Discovering Causal Signals in Images](https://mlsanity.com/paper/1605.08179/) (2016)
- [Confounding-Robust Policy Improvement](https://mlsanity.com/paper/1805.08593/) (2018)
- [Deeply-Learned Part-Aligned Representations for Person Re-Identification](https://mlsanity.com/paper/1707.07256/) (2017)
- [Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up](https://mlsanity.com/paper/1903.02827/) (2019)
- [Mixed High-Order Attention Network for Person Re-Identification](https://mlsanity.com/paper/1908.05819/) (2019)

## Built on by (most cited first, 30 of 114)

- [CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labels](https://mlsanity.com/paper/2211.13977/) (2022)
- [Dual Cross-Attention Learning for Fine-Grained Visual Categorization and Object Re-Identification](https://mlsanity.com/paper/2205.02151/) (2022)
- [Cross-view Geo-localization via Learning Disentangled Geometric Layout Correspondence](https://mlsanity.com/paper/2212.04074/) (2022)
- [CF-VLM:CounterFactual Vision-Language Fine-tuning](https://mlsanity.com/paper/2506.17267/) (2025)
- [PLIP: Language-Image Pre-training for Person Representation Learning](https://mlsanity.com/paper/2305.08386/) (2023)
- [Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification](https://mlsanity.com/paper/2403.10254/) (2024)
- [T2SG: Traffic Topology Scene Graph for Topology Reasoning in Autonomous Driving](https://mlsanity.com/paper/2411.18894/) (2024)
- [TOP-ReID: Multi-spectral Object Re-Identification with Token Permutation](https://mlsanity.com/paper/2312.09612/) (2023)
- [GaitGCI: Generative Counterfactual Intervention for Gait Recognition](https://mlsanity.com/paper/2306.03428/) (2023)
- [DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification](https://mlsanity.com/paper/2412.10650/) (2024)
- [MetaFormer: A Unified Meta Framework for Fine-Grained Recognition](https://mlsanity.com/paper/2203.02751/) (2022)
- [GeoDTR+: Toward generic cross-view geolocalization via geometric disentanglement](https://mlsanity.com/paper/2308.09624/) (2023)
- [Fine-grained Visual Classification with High-temperature Refinement and Background Suppression](https://mlsanity.com/paper/2303.06442/) (2023)
- [SR-GNN: Spatial Relation-aware Graph Neural Network for Fine-Grained Image Categorization](https://mlsanity.com/paper/2209.02109/) (2022)
- [MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt](https://mlsanity.com/paper/2412.10707/) (2024)
- [Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning](https://mlsanity.com/paper/2502.12425/) (2025)
- [SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization](https://mlsanity.com/paper/2208.14607/) (2022)
- [Counterfactual Intervention Feature Transfer for Visible-Infrared Person Re-identification](https://mlsanity.com/paper/2208.00967/) (2022)
- [Fine-grained Recognition with Learnable Semantic Data Augmentation](https://mlsanity.com/paper/2309.00399/) (2023)
- [When Large Vision-Language Models Meet Person Re-Identification](https://mlsanity.com/paper/2411.18111/) (2024)
- [Delving into Multimodal Prompting for Fine-grained Visual Classification](https://mlsanity.com/paper/2309.08912/) (2023)
- [Weakly-Supervised Video Object Grounding via Causal Intervention](https://mlsanity.com/paper/2112.00475/) (2021)
- [Advancing Fine-Grained Classification by Structure and Subject Preserving Augmentation](https://mlsanity.com/paper/2406.14551/) (2024)
- [A Novel Plug-in Module for Fine-Grained Visual Classification](https://mlsanity.com/paper/2202.03822/) (2022)
- [Prototypical Contrastive Learning-based CLIP Fine-tuning for Object Re-identification](https://mlsanity.com/paper/2310.17218/) (2023)
- [DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanations](https://mlsanity.com/paper/2311.17833/) (2023)
- [Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Data](https://mlsanity.com/paper/2210.13889/) (2022)
- [Dynamic Enhancement Network for Partial Multi-modality Person Re-identification](https://mlsanity.com/paper/2305.15762/) (2023)
- [Adaptive High-Frequency Transformer for Diverse Wildlife Re-Identification](https://mlsanity.com/paper/2410.06977/) (2024)
- [Relation Preserving Triplet Mining for Stabilising the Triplet Loss in Re-identification Systems](https://mlsanity.com/paper/2110.07933/) (2021)

## Abstract (arXiv)

Attention mechanism has demonstrated great potential in fine-grained visual recognition tasks. In this paper, we present a counterfactual attention learning method to learn more effective attention based on causal inference. Unlike most existing methods that learn visual attention based on conventional likelihood, we propose to learn the attention with counterfactual causality, which provides a tool to measure the attention quality and a powerful supervisory signal to guide the learning process. Specifically, we analyze the effect of the learned visual attention on network prediction through counterfactual intervention and maximize the effect to encourage the network to learn more useful attention for fine-grained image recognition. Empirically, we evaluate our method on a wide range of fine-grained recognition tasks where attention plays a crucial role, including fine-grained image categorization, person re-identification, and vehicle re-identification. The consistent improvement on all benchmarks demonstrates the effectiveness of our method. Code is available at https://github.com/raoyongming/CAL