# Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data

arXiv: https://arxiv.org/abs/1910.03233 · HTML: https://mlsanity.com/paper/1910.03233/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2019.

## Built on by (most cited first, 7 of 7)

- [A review on the novelty measurements of academic papers](https://mlsanity.com/paper/2501.17456/) (2025)
- [Locating the Leading Edge of Cultural Change](https://mlsanity.com/paper/2411.15068/) (2024)
- [A Content-Based Novelty Measure for Scholarly Publications: A Proof of Concept](https://mlsanity.com/paper/2401.03642/) (2024)
- [An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics](https://mlsanity.com/paper/2604.15145/) (2026)
- [Examining the Cognitive Gap Between Authors and Peer Reviewers on Academic Paper Novelty](https://mlsanity.com/paper/2606.13452/) (2026)
- [Unveiling Novelty Evolution in the field of Library and Information Science in China](https://mlsanity.com/paper/2606.29872/) (2026)
- [Measuring the Novelty of Biomedical Papers Using the Latent Distances between Knowledge Units](https://mlsanity.com/paper/2609.05175/) (2026)

## Abstract (arXiv)

Lee, Walsh, and Wang (2015) - based on Uzzi, Mukherjee, Stringer, and Jones (2013) - and Wang, Veugelers, and Stephan (2017) proposed scores based on cited references (cited journals) data which can be used to measure the novelty of papers (named as novelty scores U and W in this study). Although previous research has used novelty scores in various empirical analyses, no study has been published up to now - to the best of our knowledge - which quantitatively tested the convergent validity of novelty scores: do these scores measure what they propose to measure? Using novelty assessments by faculty members (FMs) at F1000Prime for comparison, we tested the convergent validity of the two novelty scores (U and W). FMs' assessments not only refer to the quality of biomedical papers, but also to their characteristics (by assigning certain tags to the papers): for example, are the presented findings or formulated hypotheses novel (tags "new findings" and "hypothesis")? We used these and other tags to investigate the convergent validity of both novelty scores. Our study reveals different results for the novelty scores: the results for novelty score U are mostly in agreement with previously formulated expectations. We found, for instance, that for a standard deviation (one unit) increase in novelty score U, the expected number of assignments of the "new finding" tag increase by 7.47%. The results for novelty score W, however, do not reflect convergent validity with the FMs' assessments: only the results for some tags are in agreement with the expectations. Thus, we propose - based on our results - the use of novelty score U for measuring novelty quantitatively, but question the use of novelty score W.