LSH methods for data deduplication in a Wikipedia artificial dataset
Not on the map. Why?
Abstract (arXiv)
This paper illustrates locality sensitive hasing (LSH) models for the identification and removal of nearly redundant data in a text dataset. To evaluate the different models, we create an artificial dataset for data deduplication using English Wikipedia articles. Area-Under-Curve (AUC) over 0.9 were observed for most models, with the best model reaching 0.96. Deduplication enables more effective model training by preventing the model from learning a distribution that differs from the real one as a result of the repeated data.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 8 of its 13 references that are in the map. First posted on arXiv 10 Dec 2021.
Builds on
8 of 13 references.
References · 8
- 2020Language Models are Few-Shot Learnerscited by 32,320
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
- 2018A Simple Method for Commonsense Reasoningcited by 223
- 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
- 2021A Survey on Locality Sensitive Hashing Algorithms and their Applicationscited by 40
- 2016LSH Ensemble: Internet-Scale Domain Searchcited by 24
- 2020Locality-Sensitive Hashing Scheme based on Longest Circular Co-Substringcited by 11
- 2020Experimental Analysis of Locality Sensitive Hashing Techniques for High-Dimensional Approximate Nearest Neighbor Searchescited by 4
Built on by
Not cited yet.