# #REVAL: a semantic evaluation framework for hashtag recommendation

arXiv: https://arxiv.org/abs/2305.18330 · HTML: https://mlsanity.com/paper/2305.18330/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 64 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision](https://mlsanity.com/paper/2006.15509/) (2020)
- [Antonym-Synonym Classification Based on New Sub-space Embeddings](https://mlsanity.com/paper/1906.05612/) (2019)
- [On Identifying Hashtags in Disaster Twitter Data](https://mlsanity.com/paper/2001.01323/) (2020)
- [Temporal Effects on Hashtag Reuse in Twitter: A Cognitive-Inspired Hashtag Recommendation Approach](https://mlsanity.com/paper/1701.01276/) (2017)
- [Fuzzy Approach Topic Discovery in Health and Medical Corpora](https://mlsanity.com/paper/1705.00995/) (2017)
- [Hit ratio: An Evaluation Metric for Hashtag Recommendation](https://mlsanity.com/paper/2010.01258/) (2020)
- [Medical Synonym Extraction with Concept Space Models](https://mlsanity.com/paper/1506.00528/) (2015)
- [EmTaggeR: A Word Embedding Based Novel Method for Hashtag Recommendation on Twitter](https://mlsanity.com/paper/1712.01562/) (2017)
- [Word-level Lexical Normalisation using Context-Dependent Embeddings](https://mlsanity.com/paper/1911.06172/) (2019)

## Abstract (arXiv)

Automatic evaluation of hashtag recommendation models is a fundamental task in many online social network systems. In the traditional evaluation method, the recommended hashtags from an algorithm are firstly compared with the ground truth hashtags for exact correspondences. The number of exact matches is then used to calculate the hit rate, hit ratio, precision, recall, or F1-score. This way of evaluating hashtag similarities is inadequate as it ignores the semantic correlation between the recommended and ground truth hashtags. To tackle this problem, we propose a novel semantic evaluation framework for hashtag recommendation, called #REval. This framework includes an internal module referred to as BERTag, which automatically learns the hashtag embeddings. We investigate on how the #REval framework performs under different word embedding methods and different numbers of synonyms and hashtags in the recommendation using our proposed #REval-hit-ratio measure. Our experiments of the proposed framework on three large datasets show that #REval gave more meaningful hashtag synonyms for hashtag recommendation evaluation. Our analysis also highlights the sensitivity of the framework to the word embedding technique, with #REval based on BERTag more superior over #REval based on FastText and Word2Vec.