# SCStory: Self-supervised and Continual Online Story Discovery

arXiv: https://arxiv.org/abs/2312.03725 · HTML: https://mlsanity.com/paper/2312.03725/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 61 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification](https://mlsanity.com/paper/2004.12239/) (2020)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [Towards Continual Knowledge Learning of Language Models](https://mlsanity.com/paper/2110.03215/) (2021)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [Augmenting Data with Mixup for Sentence Classification: An Empirical Study](https://mlsanity.com/paper/1905.08941/) (2019)
- [Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora](https://mlsanity.com/paper/2110.08534/) (2021)
- [A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal](https://mlsanity.com/paper/2005.10070/) (2020)
- [Knowledge-Preserving Incremental Social Event Detection via Heterogeneous GNNs](https://mlsanity.com/paper/2101.08747/) (2021)
- [Generating Representative Headlines for News Stories](https://mlsanity.com/paper/2001.09386/) (2020)
- [TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning](https://mlsanity.com/paper/2111.04198/) (2021)
- [TaxoCom: Topic Taxonomy Completion with Hierarchical Discovery of Novel Topic Clusters](https://mlsanity.com/paper/2201.06771/) (2022)
- [NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application](https://mlsanity.com/paper/2102.04887/) (2021)
- [Multilingual Clustering of Streaming News](https://mlsanity.com/paper/1809.00540/) (2018)
- [Adaptive Model Pooling for Online Deep Anomaly Detection from a Complex Evolving Data Stream](https://mlsanity.com/paper/2206.04792/) (2022)
- [Event-Driven News Stream Clustering using Entity-Aware Contextual Embeddings](https://mlsanity.com/paper/2101.11059/) (2021)
- [Batch Clustering for Multilingual News Streaming](https://mlsanity.com/paper/2004.08123/) (2020)
- [Growing Story Forest Online from Massive Breaking News](https://mlsanity.com/paper/1803.00189/) (2018)
- [Topic Taxonomy Expansion via Hierarchy-Aware Topic Phrase Generation](https://mlsanity.com/paper/2211.01981/) (2022)

## Built on by (most cited first, 5 of 5)

- [Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding](https://mlsanity.com/paper/2406.02472/) (2024)
- [Online Drift Detection with Maximum Concept Discrepancy](https://mlsanity.com/paper/2407.05375/) (2024)
- [SCTc-TE: A Comprehensive Formulation and Benchmark for Temporal Event Forecasting](https://mlsanity.com/paper/2312.01052/) (2023)
- [CREAM: Continual Retrieval on Dynamic Streaming Corpora with Adaptive Soft Memory](https://mlsanity.com/paper/2601.02708/) (2026)
- [Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streams](https://mlsanity.com/paper/2603.19250/) (2026)

## Abstract (arXiv)

We present a framework SCStory for online story discovery, that helps people digest rapidly published news article streams in real-time without human annotations. To organize news article streams into stories, existing approaches directly encode the articles and cluster them based on representation similarity. However, these methods yield noisy and inaccurate story discovery results because the generic article embeddings do not effectively reflect the story-indicative semantics in an article and cannot adapt to the rapidly evolving news article streams. SCStory employs self-supervised and continual learning with a novel idea of story-indicative adaptive modeling of news article streams. With a lightweight hierarchical embedding module that first learns sentence representations and then article representations, SCStory identifies story-relevant information of news articles and uses them to discover stories. The embedding module is continuously updated to adapt to evolving news streams with a contrastive learning objective, backed up by two unique techniques, confidence-aware memory replay and prioritized-augmentation, employed for label absence and data scarcity problems. Thorough experiments on real and the latest news data sets demonstrate that SCStory outperforms existing state-of-the-art algorithms for unsupervised online story discovery.