# Orthogonal Attention: A Cloze-Style Approach to Negation Scope Resolution

arXiv: https://arxiv.org/abs/2103.04294 · HTML: https://mlsanity.com/paper/2103.04294/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 13 references that are in the map.
- First posted on arXiv 7 Mar 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [Dynamic Coattention Networks For Question Answering](https://mlsanity.com/paper/1611.01604/) (2016)
- [Gated-Attention Readers for Text Comprehension](https://mlsanity.com/paper/1606.01549/) (2016)
- [NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolution](https://mlsanity.com/paper/1911.04211/) (2019)
- [Attention-over-Attention Neural Networks for Reading Comprehension](https://mlsanity.com/paper/1607.04423/) (2016)
- [Text Understanding with the Attention Sum Reader Network](https://mlsanity.com/paper/1603.01547/) (2016)
- [Resolving the Scope of Speculation and Negation using Transformer-Based Architectures](https://mlsanity.com/paper/2001.02885/) (2020)

## Abstract (arXiv)

Negation Scope Resolution is an extensively researched problem, which is used to locate the words affected by a negation cue in a sentence. Recent works have shown that simply finetuning transformer-based architectures yield state-of-the-art results on this task. In this work, we look at Negation Scope Resolution as a Cloze-Style task, with the sentence as the Context and the cue words as the Query. We also introduce a novel Cloze-Style Attention mechanism called Orthogonal Attention, which is inspired by Self Attention. First, we propose a framework for developing Orthogonal Attention variants, and then propose 4 Orthogonal Attention variants: OA-C, OA-CA, OA-EM, and OA-EMB. Using these Orthogonal Attention layers on top of an XLNet backbone, we outperform the finetuned XLNet state-of-the-art for Negation Scope Resolution, achieving the best results to date on all 4 datasets we experiment with: BioScope Abstracts, BioScope Full Papers, SFU Review Corpus and the *sem 2012 Dataset (Sherlock).