# Generative AI and the future of scientometrics: current topics and future questions

arXiv: https://arxiv.org/abs/2507.00783 · HTML: https://mlsanity.com/paper/2507.00783/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 99 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Generative AI](https://mlsanity.com/paper/2309.07930/) (2023)
- [Mapping the Increasing Use of LLMs in Scientific Papers](https://mlsanity.com/paper/2404.01268/) (2024)
- [DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process](https://mlsanity.com/paper/2503.08569/) (2025)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4](https://mlsanity.com/paper/2310.12321/) (2023)
- [Delving into LLM-assisted writing in biomedical publications through excess vocabulary](https://mlsanity.com/paper/2406.07016/) (2024)
- [A fine-grained comparison of pragmatic language understanding in humans and language models](https://mlsanity.com/paper/2212.06801/) (2022)
- [PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities](https://mlsanity.com/paper/2401.07078/) (2024)
- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges](https://mlsanity.com/paper/2502.12378/) (2025)
- [Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda](https://mlsanity.com/paper/2404.09554/) (2024)
- [The simulation of judgment in LLMs](https://mlsanity.com/paper/2502.04426/) (2025)
- [ChatGPT as Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictor](https://mlsanity.com/paper/2406.14765/) (2024)
- [Can ChatGPT evaluate research quality?](https://mlsanity.com/paper/2402.05519/) (2024)
- [Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models](https://mlsanity.com/paper/2408.06518/) (2024)
- [Human-LLM Coevolution: Evidence from Academic Writing](https://mlsanity.com/paper/2502.09606/) (2025)
- [Linguistic ambiguity analysis in ChatGPT](https://mlsanity.com/paper/2302.06426/) (2023)
- [Large Knowledge Model: Perspectives and Challenges](https://mlsanity.com/paper/2312.02706/) (2023)
- [Evaluating the Predictive Capacity of ChatGPT for Academic Peer Review Outcomes Across Multiple Platforms](https://mlsanity.com/paper/2411.09763/) (2024)
- [How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?](https://mlsanity.com/paper/2504.02767/) (2025)
- [A Critical Review of Large Language Models: Sensitivity, Bias, and the Path Toward Specialized AI](https://mlsanity.com/paper/2307.15425/) (2023)
- [When Large Language Models Meet Citation: A Survey](https://mlsanity.com/paper/2309.09727/) (2023)
- [Exploring the applicability of Large Language Models to citation context analysis](https://mlsanity.com/paper/2409.02443/) (2024)
- [Examining Linguistic Shifts in Academic Writing Before and After the Launch of ChatGPT: A Study on Preprint Papers](https://mlsanity.com/paper/2505.12218/) (2025)

## Built on by (most cited first, 1 of 1)

- [Structurally Human, Semantically Biased: Detecting LLM-Generated References with Embeddings and GNNs](https://mlsanity.com/paper/2601.20704/) (2026)

## Abstract (arXiv)

In this paper, we contribute to the debate on generative artificial intelligence (GenAI) in scientometrics. We argue that moving from a trial-and-error approach to an explainable and actionable use requires a principled understanding of strengths and weaknesses of GenAI as compared with other techniques and with human judgment. To this end, we introduce a conceptual framework based on the distinction between the semantic dimensions of texts, i.e. the meanings attributed to words, and their pragmatic dimension, i.e. their embedding within communicative situations. We leverage this framework to interpret the results of applications of GenAI in scientometrics and to provide guidance to users. Specifically, we conclude that key parameters to be considered are the nature of the task, the level of granularity of the analysis and whether the goal was descriptive, inferential or evaluative. These parameters lead to different strategies for using GenAI and human-machine integration. Finally, we suggest that, by generating large amounts of scientific language, GenAI might affect textual characteristics used to measure science, such as authors, words, and references. We argue that careful empirical work and theoretical reflection will be essential to remain capable of interpreting the evolving patterns of knowledge production in the age of AI.