# XRJL-HKUST at SemEval-2021 Task 4: WordNet-Enhanced Dual Multi-head Co-Attention for Reading Comprehension of Abstract Meaning

arXiv: https://arxiv.org/abs/2103.16102 · HTML: https://mlsanity.com/paper/2103.16102/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [SG-Net: Syntax-Guided Machine Reading Comprehension](https://mlsanity.com/paper/1908.05147/) (2019)
- [Gated-Attention Readers for Text Comprehension](https://mlsanity.com/paper/1606.01549/) (2016)
- [Dual Co-Matching Network for Multi-choice Reading Comprehension](https://mlsanity.com/paper/1901.09381/) (2019)
- [MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension](https://mlsanity.com/paper/1910.00458/) (2019)
- [A Co-Matching Model for Multi-choice Reading Comprehension](https://mlsanity.com/paper/1806.04068/) (2018)
- [DUMA: Reading Comprehension with Transposition Thinking](https://mlsanity.com/paper/2001.09415/) (2020)
- [Multi-task Learning with Multi-head Attention for Multi-choice Reading Comprehension](https://mlsanity.com/paper/2003.04992/) (2020)

## Built on by (most cited first, 1 of 1)

- [SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning](https://mlsanity.com/paper/2105.14879/) (2021)

## Abstract (arXiv)

This paper presents our submitted system to SemEval 2021 Task 4: Reading Comprehension of Abstract Meaning. Our system uses a large pre-trained language model as the encoder and an additional dual multi-head co-attention layer to strengthen the relationship between passages and question-answer pairs, following the current state-of-the-art model DUMA. The main difference is that we stack the passage-question and question-passage attention modules instead of calculating parallelly to simulate re-considering process. We also add a layer normalization module to improve the performance of our model. Furthermore, to incorporate our known knowledge about abstract concepts, we retrieve the definitions of candidate answers from WordNet and feed them to the model as extra inputs. Our system, called WordNet-enhanced DUal Multi-head Co-Attention (WN-DUMA), achieves 86.67% and 89.99% accuracy on the official blind test set of subtask 1 and subtask 2 respectively.