mlsanityData to 13 Oct 2026

OCHADAI-KYOTO at SemEval-2021 Task 1: Enhancing Model Generalization and Robustness for Lexical Complexity Prediction

arXiv 2105.05535 ↗First posted 12 May 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose an ensemble model for predicting the lexical complexity of words and multiword expressions (MWEs). The model receives as input a sentence with a target word or MWEand outputs its complexity score. Given that a key challenge with this task is the limited size of annotated data, our model relies on pretrained contextual representations from different state-of-the-art transformer-based language models (i.e., BERT and RoBERTa), and on a variety of training methods for further enhancing model generalization and robustness:multi-step fine-tuning and multi-task learning, and adversarial training. Additionally, we propose to enrich contextual representations by adding hand-crafted features during training. Our model achieved competitive results and ranked among the top-10 systems in both sub-tasks.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 12 May 2021.

Builds on

20 of 31 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,336
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,379
  6. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 919
  7. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 799
  8. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  9. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 294
  10. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  11. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  12. 2020Adversarial Training for Large Neural Language Modelscited by 110
8 more
  1. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 105
  2. 2018A Report on the Complex Word Identification Shared Task 2018cited by 30
  3. 2019A Surprisingly Robust Trick for Winograd Schema Challengecited by 30
  4. 2020The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understandingcited by 28
  5. 2020CompLex: A New Corpus for Lexical Complexity Prediction from Likert Scale Datacited by 24
  6. 2021Predicting Lexical Complexity in English Texts: The Complex 2.0 Datasetcited by 15
  7. 2020Adversarial Training for Commonsense Inferencecited by 13
  8. 2017Complex Word Identification: Challenges in Data Annotation and System Performancecited by 8

Built on by

Not cited yet.