# Efficient and Interpretable Transformer for Counterfactual Fairness

arXiv: https://arxiv.org/abs/2604.26188 · HTML: https://mlsanity.com/paper/2604.26188/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 29 Apr 2026.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification](https://mlsanity.com/paper/2108.08728/) (2021)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [Fairness via Representation Neutralization](https://mlsanity.com/paper/2106.12674/) (2021)
- [Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function](https://mlsanity.com/paper/1905.12801/) (2019)
- [Making Pre-trained Language Models Great on Tabular Prediction](https://mlsanity.com/paper/2403.01841/) (2024)
- [Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists](https://mlsanity.com/paper/2203.09192/) (2022)
- [Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluation](https://mlsanity.com/paper/2211.13358/) (2022)
- [Learning for Counterfactual Fairness from Observational Data](https://mlsanity.com/paper/2307.08232/) (2023)
- [Attributing Fair Decisions with Attention Interventions](https://mlsanity.com/paper/2109.03952/) (2021)
- [Robust Natural Language Understanding with Residual Attention Debiasing](https://mlsanity.com/paper/2305.17627/) (2023)
- [FairPFN: A Tabular Foundation Model for Causal Fairness](https://mlsanity.com/paper/2506.07049/) (2025)
- [Fairness-aware Vision Transformer via Debiased Self-Attention](https://mlsanity.com/paper/2301.13803/) (2023)

## Abstract (arXiv)

The growing reliance of machine learning models in high-stakes, highly regulated domains such as finance and insurance has created a growing tension between predictive performance, interpretability, and regulatory fairness requirements. In these settings, models are expected not only to deliver reliable predictions but also to provide transparent decision rationales and comply with strict fairness requirements. Attention-based transformers offer powerful mechanisms for modeling complex data relationships as demonstrated in various language tasks, yet their attention mechanisms alone do not ensure counterfactually fair predictions, even when combined with fairness-aware techniques. To address these limitations, we propose the Feature Correlation Transformer (FCorrTransformer), an attention-light architecture tailored for tabular data. In this design, the attention matrix admits a direct statistical interpretation as pairwise feature dependencies, enhancing both interpretability and efficiency. Leveraging this structure, we introduce Counterfactual Attention Regularization (CAR), a framework that enforces group-invariant fair representations of sensitive features at the attention level, promoting counterfactually fair predictions without relying on explicit causal assumptions. Empirical evaluations on imbalanced classification and regression benchmarks demonstrate that FCorrTransformer combined with CAR achieves strong counterfactual fairness while maintaining competitive predictive performance and substantially reducing model complexity compared with standard transformer-based baselines. Overall, this work bridges a critical gap between fairness theory and machine learning models, offering a practical framework for responsible AI in regulatory-sensitive domains.