Distilling BERT for low complexity network training
Not on the map. Why?
Abstract (arXiv)
This paper studies the efficiency of transferring BERT learnings to low complexity models like BiLSTM, BiLSTM with attention and shallow CNNs using sentiment analysis on SST-2 dataset. It also compares the complexity of inference of the BERT model with these lower complexity models and underlines the importance of these techniques in enabling high performance NLP models on edge devices like mobiles, tablets and MCU development boards like Raspberry Pi etc. and enabling exciting new applications.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 6 of its 10 references that are in the map. First posted on arXiv 13 May 2021.
Builds on
6 of 10 references.
References · 6
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
- 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
- 2015A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classificationcited by 92
- 2019Fine-grained Sentiment Classification using BERTcited by 38
Built on by
Not cited yet.