# Evaluating Deduplication Techniques for Economic Research Paper Titles with a Focus on Semantic Similarity using NLP and LLMs

arXiv: https://arxiv.org/abs/2410.01141 · HTML: https://mlsanity.com/paper/2410.01141/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 14 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 3 of 3)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection](https://mlsanity.com/paper/2205.10455/) (2022)
- [A Pre-trained Data Deduplication Model based on Active Learning](https://mlsanity.com/paper/2308.00721/) (2023)

## Built on by (most cited first, 1 of 1)

- [Trust & Safety of LLMs and LLMs in Trust & Safety](https://mlsanity.com/paper/2412.02113/) (2024)

## Abstract (arXiv)

This study investigates efficient deduplication techniques for a large NLP dataset of economic research paper titles. We explore various pairing methods alongside established distance measures (Levenshtein distance, cosine similarity) and a sBERT model for semantic evaluation. Our findings suggest a potentially low prevalence of duplicates based on the observed semantic similarity across different methods. Further exploration with a human-annotated ground truth set is completed for a more conclusive assessment. The result supports findings from the NLP, LLM based distance metrics.