# AttentionMix: Data augmentation method that relies on BERT attention mechanism

arXiv: https://arxiv.org/abs/2309.11104 · HTML: https://mlsanity.com/paper/2309.11104/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)
- [MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification](https://mlsanity.com/paper/2004.12239/) (2020)
- [SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization](https://mlsanity.com/paper/2006.01791/) (2020)
- [Augmenting Data with Mixup for Sentence Classification: An Empirical Study](https://mlsanity.com/paper/1905.08941/) (2019)
- [SnapMix: Semantically Proportional Mixing for Augmenting Fine-grained Data](https://mlsanity.com/paper/2012.04846/) (2020)
- [SSMix: Saliency-Based Span Mixup for Text Classification](https://mlsanity.com/paper/2106.08062/) (2021)
- [Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks](https://mlsanity.com/paper/2010.02394/) (2020)
- [The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT](https://mlsanity.com/paper/2101.09115/) (2021)
- [An overview of mixing augmentation methods and augmentation strategies](https://mlsanity.com/paper/2107.09887/) (2021)
- [How Far Does BERT Look At:Distance-based Clustering and Analysis of BERT's Attention](https://mlsanity.com/paper/2011.00943/) (2020)

## Built on by (most cited first, 2 of 2)

- [Why does Knowledge Distillation Work? Rethink its Attention and Fidelity Mechanism](https://mlsanity.com/paper/2405.00739/) (2024)
- [Enhancing elusive clues in knowledge learning by contrasting attention of language models](https://mlsanity.com/paper/2409.17954/) (2024)

## Abstract (arXiv)

The Mixup method has proven to be a powerful data augmentation technique in Computer Vision, with many successors that perform image mixing in a guided manner. One of the interesting research directions is transferring the underlying Mixup idea to other domains, e.g. Natural Language Processing (NLP). Even though there already exist several methods that apply Mixup to textual data, there is still room for new, improved approaches. In this work, we introduce AttentionMix, a novel mixing method that relies on attention-based information. While the paper focuses on the BERT attention mechanism, the proposed approach can be applied to generally any attention-based model. AttentionMix is evaluated on 3 standard sentiment classification datasets and in all three cases outperforms two benchmark approaches that utilize Mixup mechanism, as well as the vanilla BERT method. The results confirm that the attention-based information can be effectively used for data augmentation in the NLP domain.