# BigGreen at SemEval-2021 Task 1: Lexical Complexity Prediction with Assembly Models

arXiv: https://arxiv.org/abs/2104.09040 · HTML: https://mlsanity.com/paper/2104.09040/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 41 references that are in the map.
- First posted on arXiv 19 Apr 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [A Report on the Complex Word Identification Shared Task 2018](https://mlsanity.com/paper/1804.09132/) (2018)
- [CompLex: A New Corpus for Lexical Complexity Prediction from Likert Scale Data](https://mlsanity.com/paper/2003.07008/) (2020)
- [Complex Word Identification: Challenges in Data Annotation and System Performance](https://mlsanity.com/paper/1710.04989/) (2017)

## Built on by (most cited first, 2 of 2)

- [SemEval-2021 Task 1: Lexical Complexity Prediction](https://mlsanity.com/paper/2106.00473/) (2021)
- [Lexical Complexity Prediction: An Overview](https://mlsanity.com/paper/2303.04851/) (2023)

## Abstract (arXiv)

This paper describes a system submitted by team BigGreen to LCP 2021 for predicting the lexical complexity of English words in a given context. We assemble a feature engineering-based model with a deep neural network model founded on BERT. While BERT itself performs competitively, our feature engineering-based model helps in extreme cases, eg. separating instances of easy and neutral difficulty. Our handcrafted features comprise a breadth of lexical, semantic, syntactic, and novel phonological measures. Visualizations of BERT attention maps offer insight into potential features that Transformers models may learn when fine-tuned for lexical complexity prediction. Our ensembled predictions score reasonably well for the single word subtask, and we demonstrate how they can be harnessed to perform well on the multi word expression subtask too.