# Evaluation of Thematic Coherence in Microblogs

arXiv: https://arxiv.org/abs/2106.15971 · HTML: https://mlsanity.com/paper/2106.15971/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 53 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 10 of 10)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence](https://mlsanity.com/paper/2004.03974/) (2020)
- [Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads](https://mlsanity.com/paper/1511.07487/) (2015)
- [Topic Modeling with Wasserstein Autoencoders](https://mlsanity.com/paper/1907.12374/) (2019)
- [Improving Topic Models with Latent Feature Word Representations](https://mlsanity.com/paper/1810.06306/) (2018)
- [Supporting the use of user generated content in journalistic practice](https://mlsanity.com/paper/1702.06491/) (2017)

## Built on by (most cited first, 3 of 3)

- [Template-based Abstractive Microblog Opinion Summarisation](https://mlsanity.com/paper/2208.04083/) (2022)
- [Disentangled Learning of Stance and Aspect Topics for Vaccine Attitude Detection in Social Media](https://mlsanity.com/paper/2205.03296/) (2022)
- [Unsupervised Opinion Summarisation in the Wasserstein Space](https://mlsanity.com/paper/2211.14923/) (2022)

## Abstract (arXiv)

Collecting together microblogs representing opinions about the same topics within the same timeframe is useful to a number of different tasks and practitioners. A major question is how to evaluate the quality of such thematic clusters. Here we create a corpus of microblog clusters from three different domains and time windows and define the task of evaluating thematic coherence. We provide annotation guidelines and human annotations of thematic coherence by journalist experts. We subsequently investigate the efficacy of different automated evaluation metrics for the task. We consider a range of metrics including surface level metrics, ones for topic model coherence and text generation metrics (TGMs). While surface level metrics perform well, outperforming topic coherence metrics, they are not as consistent as TGMs. TGMs are more reliable than all other metrics considered for capturing thematic coherence in microblog clusters due to being less sensitive to the effect of time windows.