# Does a Large Language Model Really Speak in Human-Like Language?

arXiv: https://arxiv.org/abs/2501.01273 · HTML: https://mlsanity.com/paper/2501.01273/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Builds on (most cited first, 12 of 12)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners](https://mlsanity.com/paper/2406.11050/) (2024)
- [Knowledge of cultural moral norms in large language models](https://mlsanity.com/paper/2306.01857/) (2023)
- [GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Models](https://mlsanity.com/paper/2310.00737/) (2023)
- [Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets](https://mlsanity.com/paper/1709.09500/) (2017)
- [DEBATE: Devil's Advocate-Based Assessment and Text Evaluation](https://mlsanity.com/paper/2405.09935/) (2024)
- [Statistically significant detection of semantic shifts using contextual word embeddings](https://mlsanity.com/paper/2104.03776/) (2021)
- [Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Study](https://mlsanity.com/paper/2402.15518/) (2024)
- [Statistical Significance Testing in Information Retrieval: An Empirical Analysis of Type I, Type II and Type III Errors](https://mlsanity.com/paper/1905.11096/) (2019)
- [Two halves of a meaningful text are statistically different](https://mlsanity.com/paper/2004.06474/) (2020)

## Built on by (most cited first, 1 of 1)

- [Comparing LLM-generated and human-authored news text using formal syntactic theory](https://mlsanity.com/paper/2506.01407/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have recently emerged, attracting considerable attention due to their ability to generate highly natural, human-like text. This study compares the latent community structures of LLM-generated text and human-written text within a hypothesis testing procedure. Specifically, we analyze three text sets: original human-written texts (O), their LLM-paraphrased versions (G), and a twice-paraphrased set (S) derived from G. Our analysis addresses two key questions: (1) Is the difference in latent community structures between O and G the same as that between G and S? (2) Does G become more similar to O as the LLM parameter controlling text variability is adjusted? The first question is based on the assumption that if LLM-generated text truly resembles human language, then the gap between the pair (O, G) should be similar to that between the pair (G, S), as both pairs consist of an original text and its paraphrase. The second question examines whether the degree of similarity between LLM-generated and human text varies with changes in the breadth of text generation. To address these questions, we propose a statistical hypothesis testing framework that leverages the fact that each text has corresponding parts across all datasets due to their paraphrasing relationship. This relationship enables the mapping of one dataset's relative position to another, allowing two datasets to be mapped to a third dataset. As a result, both mapped datasets can be quantified with respect to the space characterized by the third dataset, facilitating a direct comparison between them. Our results indicate that GPT-generated text remains distinct from human-authored text.