# Story Ending Prediction by Transferable BERT

arXiv: https://arxiv.org/abs/1905.07504 · HTML: https://mlsanity.com/paper/1905.07504/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2019.

## Built on by (most cited first, 10 of 10)

- [It's not Rocket Science : Interpreting Figurative Language in Narratives](https://mlsanity.com/paper/2109.00087/) (2021)
- [When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks](https://mlsanity.com/paper/2311.08993/) (2023)
- [Guided Generation of Cause and Effect](https://mlsanity.com/paper/2107.09846/) (2021)
- [CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision](https://mlsanity.com/paper/2107.09852/) (2021)
- [EventGround: Narrative Reasoning by Grounding to Eventuality-centric Knowledge Graphs](https://mlsanity.com/paper/2404.00209/) (2024)
- [Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis](https://mlsanity.com/paper/2210.09803/) (2022)
- [A Graph Enhanced BERT Model for Event Prediction](https://mlsanity.com/paper/2205.10822/) (2022)
- [Transferring Procedural Knowledge across Commonsense Tasks](https://mlsanity.com/paper/2304.13867/) (2023)
- [Sentence Semantic Regression for Text Generation](https://mlsanity.com/paper/2108.02984/) (2021)
- [A-CAP: Anticipation Captioning with Commonsense Knowledge](https://mlsanity.com/paper/2304.06602/) (2023)

## Abstract (arXiv)

Recent advances, such as GPT and BERT, have shown success in incorporating a pre-trained transformer language model and fine-tuning operation to improve downstream NLP systems. However, this framework still has some fundamental problems in effectively incorporating supervised knowledge from other related tasks. In this study, we investigate a transferable BERT (TransBERT) training framework, which can transfer not only general language knowledge from large-scale unlabeled data but also specific kinds of knowledge from various semantically related supervised tasks, for a target task. Particularly, we propose utilizing three kinds of transfer tasks, including natural language inference, sentiment classification, and next action prediction, to further train BERT based on a pre-trained model. This enables the model to get a better initialization for the target task. We take story ending prediction as the target task to conduct experiments. The final result, an accuracy of 91.8%, dramatically outperforms previous state-of-the-art baseline methods. Several comparative experiments give some helpful suggestions on how to select transfer tasks. Error analysis shows what are the strength and weakness of BERT-based models for story ending prediction.