# Investigating Text Shortening Strategy in BERT: Truncation vs Summarization

arXiv: https://arxiv.org/abs/2403.12799 · HTML: https://mlsanity.com/paper/2403.12799/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 17 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding](https://mlsanity.com/paper/2009.05387/) (2020)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- [IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP](https://mlsanity.com/paper/2011.00677/) (2020)
- [IndoSum: A New Benchmark Dataset for Indonesian Text Summarization](https://mlsanity.com/paper/1810.05334/) (2018)
- [Liputan6: A Large-scale Indonesian Dataset for Text Summarization](https://mlsanity.com/paper/2011.00679/) (2020)
- [Fine-tuning Pretrained Multilingual BERT Model for Indonesian Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2103.03732/) (2021)

## Built on by (most cited first, 2 of 2)

- [Exploration of Summarization by Generative Language Models for Automated Scoring of Long Essays](https://mlsanity.com/paper/2510.22830/) (2025)
- [DisCEdge: Distributed Context Management for Large Language Models at the Edge](https://mlsanity.com/paper/2511.22599/) (2025)

## Abstract (arXiv)

The parallelism of Transformer-based models comes at the cost of their input max-length. Some studies proposed methods to overcome this limitation, but none of them reported the effectiveness of summarization as an alternative. In this study, we investigate the performance of document truncation and summarization in text classification tasks. Each of the two was investigated with several variations. This study also investigated how close their performances are to the performance of full-text. We used a dataset of summarization tasks based on Indonesian news articles (IndoSum) to do classification tests. This study shows how the summaries outperform the majority of truncation method variations and lose to only one. The best strategy obtained in this study is taking the head of the document. The second is extractive summarization. This study explains what happened to the result, leading to further research in order to exploit the potential of document summarization as a shortening alternative. The code and data used in this work are publicly available in https://github.com/mirzaalimm/TruncationVsSummarization.