FairNVT: Fair Classification via Noise Injection in Vision Transformers

This paper presents FairNVT, a lightweight debiasing framework for pretrained transformer-based encoders that improves prediction fairness while preserving task performance. FairNVT is motivated by the intuition that reducing sensitive-attribute information in the representation used by the downstream classifier can facilitate fairer predictions. Our approach learns task-relevant and sensitive embeddings via lightweight adapters, applies calibrated Gaussian noise to the sensitive embedding, and fuses it with the task representation. Together with orthogonality constraints and fairness regularization, these components jointly reduce sensitive-attribute leakage in the learned embeddings and encourage fairer downstream predictions. Across three datasets spanning vision and language, FairNVT reduces sensitive-attribute attacker accuracy, improves fairness metrics such as demographic parity difference and equalized odds, and maintains competitive task performance.