A Split-then-Join Approach to Abstractive Summarization for Very Long Documents in a Low Resource Setting

textttBIGBIRD-PEGASUS model achieves textitstate-of-the-art on abstractive text summarization for long documents. However it's capacity still limited to maximum of 4,096 tokens, thus caused performance degradation on summarization for very long documents. Common method to deal with the issue is to truncate the documents. In this reasearch, we'll use different approach. We'll use the pretrained textttBIGBIRD-PEGASUS model by fine tuned the model on other domain dataset. First, we filter out all documents which length less than 20,000 tokens to focus on very long documents. To prevent domain shifting problem and overfitting on transfer learning due to small dataset, we augment the dataset by splitting document-summary training pair into parts, to fit the document into 4,096 tokens. Source code available on hrefhttps://github.com/lhfazry/SPIN-summhttps://github.com/lhfazry/SPIN-summ.