mlsanityData to 14 Oct 2026

MOROCCO: Model Resource Comparison Framework

arXiv 2104.14314 ↗First posted 29 Apr 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The new generation of pre-trained NLP models push the SOTA to the new limits, but at the cost of computational resources, to the point that their use in real production environments is often prohibitively expensive. We tackle this problem by evaluating not only the standard quality metrics on downstream tasks but also the memory footprint and inference time. We present MOROCCO, a framework to compare language models compatible with textttjiant environment which supports over 50 NLU tasks, including SuperGLUE benchmark and multiple probing suites. We demonstrate its applicability for two GLUE-like suites in different languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 29 Apr 2021.

Builds on

19 of 24 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  8. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  10. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  11. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 249
  12. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
7 more
  1. 2019Adaptation of Deep Bidirectional Multilingual Transformers for Russian Languagecited by 109
  2. 2020DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialoguecited by 89
  3. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  4. 2019Small and Practical BERT Models for Sequence Labelingcited by 40
  5. 2020jiant: A Software Toolkit for Research on General-Purpose Text Understanding Modelscited by 39
  6. 2019Findings of the Third Workshop on Neural Generation and Translationcited by 8
  7. 2018Findings of the Second Workshop on Neural Machine Translation and Generationcited by 5

Built on by

Not cited yet.