# SkIn: Skimming-Intensive Long-Text Classification Using BERT for Medical Corpus

arXiv: https://arxiv.org/abs/2209.05741 · HTML: https://mlsanity.com/paper/2209.05741/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 13 Sep 2022.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [BERT: A Review of Applications in Natural Language Processing and Understanding](https://mlsanity.com/paper/2103.11943/) (2021)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- [Dropout as data augmentation](https://mlsanity.com/paper/1506.08700/) (2015)
- [Efficient and Robust Question Answering from Minimal Context over Documents](https://mlsanity.com/paper/1805.08092/) (2018)
- [Phenotyping of Clinical Notes with Improved Document Classification Models Using Contextualized Neural Language Models](https://mlsanity.com/paper/1910.13664/) (2019)
- [Text Guide: Improving the quality of long text classification by a text selection method based on feature importance](https://mlsanity.com/paper/2104.07225/) (2021)

## Abstract (arXiv)

BERT is a widely used pre-trained model in natural language processing. However, since BERT is quadratic to the text length, the BERT model is difficult to be used directly on the long-text corpus. In some fields, the collected text data may be quite long, such as in the health care field. Therefore, to apply the pre-trained language knowledge of BERT to long text, in this paper, imitating the skimming-intensive reading method used by humans when reading a long paragraph, the Skimming-Intensive Model (SkIn) is proposed. It can dynamically select the critical information in the text so that the sentence input into the BERT-Base model is significantly shortened, which can effectively save the cost of the classification algorithm. Experiments show that the SkIn method has achieved superior accuracy than the baselines on long-text classification datasets in the medical field, while its time and space requirements increase linearly with the text length, alleviating the time and space overflow problem of basic BERT on long-text data.