# Empowering Vision Transformers with Multi-Scale Causal Intervention for Long-Tailed Image Classification

arXiv: https://arxiv.org/abs/2505.08173 · HTML: https://mlsanity.com/paper/2505.08173/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 57 references that are in the map.
- First posted on arXiv 13 May 2025.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Balanced Meta-Softmax for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2007.10740/) (2020)
- [Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect](https://mlsanity.com/paper/2009.12991/) (2020)
- [Disentangling Label Distribution for Long-tailed Visual Recognition](https://mlsanity.com/paper/2012.00321/) (2020)
- [Causality Inspired Representation Learning for Domain Generalization](https://mlsanity.com/paper/2203.14237/) (2022)
- [Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification](https://mlsanity.com/paper/2103.14267/) (2021)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [ResLT: Residual Learning for Long-tailed Recognition](https://mlsanity.com/paper/2101.10633/) (2021)
- [Causal Attention for Unbiased Visual Recognition](https://mlsanity.com/paper/2108.08782/) (2021)
- [LPT: Long-tailed Prompt Tuning for Image Classification](https://mlsanity.com/paper/2210.01033/) (2022)
- [VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2111.13579/) (2021)
- [Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering](https://mlsanity.com/paper/2207.12647/) (2022)
- [Vision-and-Language Navigation via Causal Learning](https://mlsanity.com/paper/2404.10241/) (2024)
- [Generalized Parametric Contrastive Learning](https://mlsanity.com/paper/2209.12400/) (2022)
- [Global and Local Mixture Consistency Cumulative Learning for Long-tailed Visual Recognitions](https://mlsanity.com/paper/2305.08661/) (2023)
- [Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2308.03457/) (2023)
- [Learning Imbalanced Data with Vision Transformers](https://mlsanity.com/paper/2212.02015/) (2022)
- [Context De-confounded Emotion Recognition](https://mlsanity.com/paper/2303.11921/) (2023)
- [Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Models](https://mlsanity.com/paper/2310.08106/) (2023)
- [CUDA: Curriculum of Data Augmentation for Long-Tailed Recognition](https://mlsanity.com/paper/2302.05499/) (2023)

## Built on by (most cited first, 2 of 2)

- [Class-wise Balancing Data Replay for Federated Class-Incremental Learning](https://mlsanity.com/paper/2507.07712/) (2025)
- [From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneity](https://mlsanity.com/paper/2604.08617/) (2026)

## Abstract (arXiv)

Causal inference has emerged as a promising approach to mitigate long-tail classification by handling the biases introduced by class imbalance. However, along with the change of advanced backbone models from Convolutional Neural Networks (CNNs) to Visual Transformers (ViT), existing causal models may not achieve an expected performance gain. This paper investigates the influence of existing causal models on CNNs and ViT variants, highlighting that ViT's global feature representation makes it hard for causal methods to model associations between fine-grained features and predictions, which leads to difficulties in classifying tail classes with similar visual appearance. To address these issues, this paper proposes TSCNet, a two-stage causal modeling method to discover fine-grained causal associations through multi-scale causal interventions. Specifically, in the hierarchical causal representation learning stage (HCRL), it decouples the background and objects, applying backdoor interventions at both the patch and feature level to prevent model from using class-irrelevant areas to infer labels which enhances fine-grained causal representation. In the counterfactual logits bias calibration stage (CLBC), it refines the optimization of model's decision boundary by adaptive constructing counterfactual balanced data distribution to remove the spurious associations in the logits caused by data distribution. Extensive experiments conducted on various long-tail benchmarks demonstrate that the proposed TSCNet can eliminate multiple biases introduced by data imbalance, which outperforms existing methods.