# Your Large Language Models Are Leaving Fingerprints

arXiv: https://arxiv.org/abs/2405.14057 · HTML: https://mlsanity.com/paper/2405.14057/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 14 of 14)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://mlsanity.com/paper/2310.05130/) (2023)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://mlsanity.com/paper/2305.14902/) (2023)
- [The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance](https://mlsanity.com/paper/2401.03729/) (2024)
- [OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples](https://mlsanity.com/paper/2307.11729/) (2023)
- [Evade ChatGPT Detectors via A Single Space](https://mlsanity.com/paper/2307.02599/) (2023)
- [Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis](https://mlsanity.com/paper/2304.05534/) (2023)

## Built on by (most cited first, 30 of 33)

- [REEF: Representation Encoding Fingerprints for Large Language Models](https://mlsanity.com/paper/2410.14273/) (2024)
- [Idiosyncrasies in Large Language Models](https://mlsanity.com/paper/2502.12150/) (2025)
- [SoK: Large Language Model Copyright Auditing via Fingerprinting](https://mlsanity.com/paper/2508.19843/) (2025)
- [Model Provenance Testing for Large Language Models](https://mlsanity.com/paper/2502.00706/) (2025)
- [Gradient-Based Model Fingerprinting for LLM Similarity Detection and Family Classification](https://mlsanity.com/paper/2506.01631/) (2025)
- [Detecting Stylistic Fingerprints of Large Language Models](https://mlsanity.com/paper/2503.01659/) (2025)
- [MoSEs: Uncertainty-Aware AI-Generated Text Detection via Mixture of Stylistics Experts with Conditional Thresholds](https://mlsanity.com/paper/2509.02499/) (2025)
- [StoryScope: Investigating idiosyncrasies in AI fiction](https://mlsanity.com/paper/2604.03136/) (2026)
- [LLMs Have Rhythm: Fingerprinting Large Language Models Using Inter-Token Times and Network Traffic Analysis](https://mlsanity.com/paper/2502.20589/) (2025)
- [Are Robust LLM Fingerprints Adversarially Robust?](https://mlsanity.com/paper/2509.26598/) (2025)
- [Real Money, Fake Models: Deceptive Model Claims in Shadow APIs](https://mlsanity.com/paper/2603.01919/) (2026)
- [Hardware and Software Platform Inference](https://mlsanity.com/paper/2411.05197/) (2024)
- [Invisible Traces: Using Hybrid Fingerprinting to identify underlying LLMs in GenAI Apps](https://mlsanity.com/paper/2501.18712/) (2025)
- [Peering Behind the Shield: Guardrail Identification in Large Language Models](https://mlsanity.com/paper/2502.01241/) (2025)
- [The Challenge of Identifying the Origin of Black-Box Large Language Models](https://mlsanity.com/paper/2503.04332/) (2025)
- [Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs](https://mlsanity.com/paper/2510.05148/) (2025)
- [Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimization](https://mlsanity.com/paper/2604.15022/) (2026)
- [AWM: Accurate Weight-Matrix Fingerprint for Large Language Models](https://mlsanity.com/paper/2510.06738/) (2025)
- [On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?](https://mlsanity.com/paper/2510.20810/) (2025)
- [The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive](https://mlsanity.com/paper/2604.25634/) (2026)
- [Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Services](https://mlsanity.com/paper/2606.16100/) (2026)
- [From Construction to Injection: Edit-Based Fingerprints for Large Language Models](https://mlsanity.com/paper/2509.03122/) (2025)
- [Asymmetric Idiosyncrasies in Multimodal Models](https://mlsanity.com/paper/2602.22734/) (2026)
- [Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Papers](https://mlsanity.com/paper/2603.25638/) (2026)
- [BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems](https://mlsanity.com/paper/2606.20668/) (2026)
- [Construction-Driven Injection: Linguistically-Grounded Edit-Based Code-Mixing Fingerprints for Large Language Models](https://mlsanity.com/paper/2607.25633/) (2026)
- [FNF: Functional Network Fingerprint for Large Language Models](https://mlsanity.com/paper/2601.22692/) (2026)
- [PragLocker: Protecting Agent Intellectual Property in Untrusted Deployments via Non-Portable Prompts](https://mlsanity.com/paper/2605.05974/) (2026)
- [T5-CSBoost: Adversarial Perturbation Resistant LLM Fingerprinting](https://mlsanity.com/paper/2607.14113/) (2026)
- [AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes](https://mlsanity.com/paper/2609.00052/) (2026)

## Abstract (arXiv)

It has been shown that finetuned transformers and other supervised detectors effectively distinguish between human and machine-generated text in some situations arXiv:2305.13242, but we find that even simple classifiers on top of n-gram and part-of-speech features can achieve very robust performance on both in- and out-of-domain data. To understand how this is possible, we analyze machine-generated output text in five datasets, finding that LLMs possess unique fingerprints that manifest as slight differences in the frequency of certain lexical and morphosyntactic features. We show how to visualize such fingerprints, describe how they can be used to detect machine-generated text and find that they are even robust across textual domains. We find that fingerprints are often persistent across models in the same model family (e.g. llama-13b vs. llama-65b) and that models fine-tuned for chat are easier to detect than standard language models, indicating that LLM fingerprints may be directly induced by the training data.