# Calibrating Transformer Attention via Task-Space Sensitivity Feedback

arXiv: https://arxiv.org/abs/2512.20661 · HTML: https://mlsanity.com/paper/2512.20661/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 19 Dec 2025.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [When Attention Sink Emerges in Language Models: An Empirical View](https://mlsanity.com/paper/2410.10781/) (2024)
- [Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration](https://mlsanity.com/paper/2406.15765/) (2024)
- [Deriving Machine Attention from Human Rationales](https://mlsanity.com/paper/1808.09367/) (2018)
- [Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Mining](https://mlsanity.com/paper/1808.00265/) (2018)
- [Think Clearly: Improving Reasoning via Redundant Token Pruning](https://mlsanity.com/paper/2507.08806/) (2025)
- [Attend to Who You Are: Supervising Self-Attention for Keypoint Detection and Instance-Aware Association](https://mlsanity.com/paper/2111.12892/) (2021)
- [Aligning Human and Machine Attention for Enhanced Supervised Learning](https://mlsanity.com/paper/2502.06811/) (2025)

## Abstract (arXiv)

Transformer-based pre-trained language models (PLMs) excel in text classification but suffer from attention dilution and attention sink effects, forcing models to over-focus on task-irrelevant tokens. Existing attention supervision methods rely on costly token-level human annotations or static heuristics, which fail to scale or capture context-dependent token importance. To address this, we propose AttCal, a self-supervised, annotation-free attention calibration framework via task-space sensitivity feedback. AttCal treats attention distributions as stochastic policies to guide token deletion perturbations, infers context-dependent token importance from semantic shifts, and optimizes attention parameters via policy gradient. Critically, AttCal freezes the PLM backbone and updates only attention projection layers to ensure training efficiency. Benchmarks across five datasets demonstrate that AttCal significantly outperforms vanilla PLMs-yielding up to a 15.8% accuracy lift for Llama3-8B-and surpasses SOTA supervised baselines in both classification and interpretability.