# Merlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference

arXiv: https://arxiv.org/abs/2605.09990 · HTML: https://mlsanity.com/paper/2605.09990/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 11 May 2026.

## Builds on (most cited first, 14 of 14)

- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [D4: Improving LLM Pretraining via Document De-Duplication and Diversification](https://mlsanity.com/paper/2308.12284/) (2023)
- [Extracting books from production language models](https://mlsanity.com/paper/2601.02671/) (2026)
- [SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training](https://mlsanity.com/paper/2407.06654/) (2024)
- [The Mosaic Memory of Large Language Models](https://mlsanity.com/paper/2405.15523/) (2024)
- [REFRAG: Rethinking RAG based Decoding](https://mlsanity.com/paper/2509.01092/) (2025)
- [Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference](https://mlsanity.com/paper/2604.02985/) (2026)
- [LSHBloom: Memory-efficient, Extreme-scale Document Deduplication](https://mlsanity.com/paper/2411.04257/) (2024)
- [ZipLLM: Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression](https://mlsanity.com/paper/2505.06252/) (2025)
- [Rethinking Soft Compression in Retrieval-Augmented Generation: A Query-Conditioned Selector Perspective](https://mlsanity.com/paper/2602.15856/) (2026)
- [Accelerating Data Chunking in Deduplication Systems using Vector Instructions](https://mlsanity.com/paper/2508.05797/) (2025)
- [Tidehunter: Large-Value Storage With Minimal Data Relocation](https://mlsanity.com/paper/2602.01873/) (2026)

## Built on by (most cited first, 2 of 2)

- [A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forever](https://mlsanity.com/paper/2607.23806/) (2026)
- [Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost](https://mlsanity.com/paper/2609.39358/) (2026)

## Abstract (arXiv)

Data-intensive applications, ranging from large-scale retrieval systems to advanced data pipelines, are increasingly bottlenecked by the processing of highly redundant text corpora. We present Merlin, a local-first, agnostic, high-throughput deduplication and context optimization engine designed to mitigate these inefficiencies. Utilizing a highly optimized, SIMD-friendly open-addressing flat hash set combined with xxHash3-64, Merlin performs rapid, byte-exact deduplication of text passages and data chunks. While broadly applicable to any text-processing workflow, its impact is particularly pronounced in Large Language Model (LLM) ecosystems, such as Retrieval-Augmented Generation (RAG). Our empirical evaluations demonstrate an input reduction ranging from 13.9% in low-redundancy datasets to over 71% in high-redundancy pipelines, maintaining absolute data fidelity. Furthermore, we detail the system's integration architecture via the Model Context Protocol (MCP), enabling secure, zero-network-interception deployment across major IDEs and autonomous agents. This paper outlines the core algorithmic design, performance benchmarks, and the architectural principles required to process data at sustained speeds of up to 8.7 GB/s.