# Artificial intelligence technologies to support research assessment: A review arXiv: https://arxiv.org/abs/2212.06574 ยท HTML: https://mlsanity.com/paper/2212.06574/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 58 of its 358 references that are in the map. - First posted on arXiv 11 Dec 2022. ## Builds on (most cited first, 30 of 58) - [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019) - [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018) - [Explainable Artificial Intelligence: a Systematic Review](https://mlsanity.com/paper/2006.00093/) (2020) - [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021) - [Aspect-based Sentiment Analysis of Scientific Reviews](https://mlsanity.com/paper/2006.03257/) (2020) - [An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process](https://mlsanity.com/paper/2010.05137/) (2020) - [Predicting citation counts based on deep neural network learning techniques](https://mlsanity.com/paper/1809.04365/) (2018) - [Tortured phrases: A dubious writing style emerging in science. Evidence of critical issues affecting established journals](https://mlsanity.com/paper/2107.06751/) (2021) - [OpenCitations, an infrastructure organization for open scholarship](https://mlsanity.com/paper/1906.11964/) (2019) - [Viziometrics: Analyzing Visual Information in the Scientific Literature](https://mlsanity.com/paper/1605.04951/) (2016) - [The Journal Coverage of Web of Science and Scopus: a Comparative Analysis](https://mlsanity.com/paper/1511.08096/) (2015) - [Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations' COCI: a multidisciplinary comparison of coverage via citations](https://mlsanity.com/paper/2004.14329/) (2020) - [Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic](https://mlsanity.com/paper/2005.10732/) (2020) - [Structured Citation Trend Prediction Using Graph Neural Networks](https://mlsanity.com/paper/2104.02562/) (2021) - [Large Scale Subject Category Classification of Scholarly Papers with Deep Attentive Neural Networks](https://mlsanity.com/paper/2007.13826/) (2020) - [The Journal Coverage of Web of Science, Scopus and Dimensions: A Comparative Analysis](https://mlsanity.com/paper/2011.00223/) (2020) - [Team size matters: Collaboration and scientific impact since 1900](https://mlsanity.com/paper/1410.8544/) (2014) - [International Research Collaboration: Novelty, Conventionality, and Atypicality in Knowledge Recombination](https://mlsanity.com/paper/1804.09070/) (2018) - [Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective](https://mlsanity.com/paper/1401.4321/) (2014) - [Could scientists use Altmetric.com scores to predict longer term citation counts?](https://mlsanity.com/paper/1801.10311/) (2018) - [Analyzing Linguistic Complexity and Scientific Impact](https://mlsanity.com/paper/1907.11843/) (2019) - [Early Indicators of Scientific Impact: Predicting Citations with Altmetrics](https://mlsanity.com/paper/2012.13599/) (2020) - [Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge?](https://mlsanity.com/paper/1511.05078/) (2015) - [Scholarly use of social media and altmetrics: a review of the literature](https://mlsanity.com/paper/1608.08112/) (2016) - [Citation-based clustering of publications using CitNetExplorer and VOSviewer](https://mlsanity.com/paper/1702.03411/) (2017) - [Impact of lexical and sentiment factors on the popularity of scientific papers](https://mlsanity.com/paper/1605.07465/) (2016) - [Three practical field normalised alternative indicator formulae for research evaluation](https://mlsanity.com/paper/1612.01431/) (2016) - [Dimensions: A Competitor to Scopus and the Web of Science?](https://mlsanity.com/paper/1803.07525/) (2018) - [Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK Research Excellence Framework (REF)](https://mlsanity.com/paper/1807.03977/) (2018) - [The Operationalization of "Fields" as WoS Subject Categories (WCs) in Evaluative Bibliometrics: The cases of "Library and Information Science" and "Science & Technology Studies"](https://mlsanity.com/paper/1407.7849/) (2014) ## Built on by (most cited first, 1 of 1) - [Speculative Exploration on the Concept of Artificial Agents Conducting Autonomous Research](https://mlsanity.com/paper/2312.03497/) (2023) ## Abstract (arXiv) This literature review identifies indicators that associate with higher impact or higher quality research from article text (e.g., titles, abstracts, lengths, cited references and readability) or metadata (e.g., the number of authors, international or domestic collaborations, journal impact factors and authors' h-index). This includes studies that used machine learning techniques to predict citation counts or quality scores for journal articles or conference papers. The literature review also includes evidence about the strength of association between bibliometric indicators and quality score rankings from previous UK Research Assessment Exercises (RAEs) and REFs in different subjects and years and similar evidence from other countries (e.g., Australia and Italy). In support of this, the document also surveys studies that used public datasets of citations, social media indictors or open review texts (e.g., Dimensions, OpenCitations, Altmetric.com and Publons) to help predict the scholarly impact of articles. The results of this part of the literature review were used to inform the experiments using machine learning to predict REF journal article quality scores, as reported in the AI experiments report for this project. The literature review also covers technology to automate editorial processes, to provide quality control for papers and reviewers' suggestions, to match reviewers with articles, and to automatically categorise journal articles into fields. Bias and transparency in technology assisted assessment are also discussed.