# Guiding Attention for Self-Supervised Learning with Transformers

arXiv: https://arxiv.org/abs/2010.02399 · HTML: https://mlsanity.com/paper/2010.02399/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Pushdown Layers: Encoding Recursive Structure in Transformer Language Models](https://mlsanity.com/paper/2310.19089/) (2023)
- [Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems](https://mlsanity.com/paper/2310.12956/) (2023)
- [Retroformer: Pushing the Limits of Interpretable End-to-end Retrosynthesis Transformer](https://mlsanity.com/paper/2201.12475/) (2022)
- [Concept-Centric Transformers: Enhancing Model Interpretability through Object-Centric Concept Learning within a Shared Global Workspace](https://mlsanity.com/paper/2305.15775/) (2023)
- [Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding](https://mlsanity.com/paper/2204.02922/) (2022)
- [MUX-PLMs: Data Multiplexing for High-throughput Language Models](https://mlsanity.com/paper/2302.12441/) (2023)
- [Improving Speech Emotion Recognition Through Focus and Calibration Attention Mechanisms](https://mlsanity.com/paper/2208.10491/) (2022)
- [Sneaking Syntax into Transformer Language Models with Tree Regularization](https://mlsanity.com/paper/2411.18885/) (2024)
- [Beyond Self-learned Attention: Mitigating Attention Bias in Transformer-based Models Using Attention Guidance](https://mlsanity.com/paper/2402.16790/) (2024)
- [GiLT: Augmenting Transformer Language Models with Dependency Graphs](https://mlsanity.com/paper/2605.15562/) (2026)
- [Cross-lingual AMR Aligner: Paying Attention to Cross-Attention](https://mlsanity.com/paper/2206.07587/) (2022)
- [ConfRetro: a 3D-aware template-free method for enhancing retrosynthesis via molecular conformer information](https://mlsanity.com/paper/2501.12434/) (2025)
- [Copy-Augmented Representation for Structure Invariant Template-Free Retrosynthesis](https://mlsanity.com/paper/2510.16588/) (2025)

## Abstract (arXiv)

In this paper, we propose a simple and effective technique to allow for efficient self-supervised learning with bi-directional Transformers. Our approach is motivated by recent studies demonstrating that self-attention patterns in trained models contain a majority of non-linguistic regularities. We propose a computationally efficient auxiliary loss function to guide attention heads to conform to such patterns. Our method is agnostic to the actual pre-training objective and results in faster convergence of models as well as better performance on downstream tasks compared to the baselines, achieving state of the art results in low-resource settings. Surprisingly, we also find that linguistic properties of attention heads are not necessarily correlated with language modeling performance.