# ReCAM@IITK at SemEval-2021 Task 4: BERT and ALBERT based Ensemble for Abstract Word Prediction

arXiv: https://arxiv.org/abs/2104.01563 · HTML: https://mlsanity.com/paper/2104.01563/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 4 Apr 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations](https://mlsanity.com/paper/1511.02301/) (2015)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- [Gated-Attention Readers for Text Comprehension](https://mlsanity.com/paper/1606.01549/) (2016)
- [MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge](https://mlsanity.com/paper/1803.05223/) (2018)
- [Reasoning with Memory Augmented Neural Networks for Language Comprehension](https://mlsanity.com/paper/1610.06454/) (2016)

## Abstract (arXiv)

This paper describes our system for Task 4 of SemEval-2021: Reading Comprehension of Abstract Meaning (ReCAM). We participated in all subtasks where the main goal was to predict an abstract word missing from a statement. We fine-tuned the pre-trained masked language models namely BERT and ALBERT and used an Ensemble of these as our submitted system on Subtask 1 (ReCAM-Imperceptibility) and Subtask 2 (ReCAM-Nonspecificity). For Subtask 3 (ReCAM-Intersection), we submitted the ALBERT model as it gives the best results. We tried multiple approaches and found that Masked Language Modeling(MLM) based approach works the best.