# LinkTransformer: A Unified Package for Record Linkage with Transformer Language Models

arXiv: https://arxiv.org/abs/2309.00789 · HTML: https://mlsanity.com/paper/2309.00789/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 2 Sep 2023.

## Builds on (most cited first, 14 of 14)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Deep Entity Matching with Pre-Trained Language Models](https://mlsanity.com/paper/2004.00584/) (2020)
- [Neural Networks for Entity Matching: A Survey](https://mlsanity.com/paper/2010.11075/) (2020)
- [(Almost) All of Entity Resolution](https://mlsanity.com/paper/2008.04443/) (2020)
- [Using ChatGPT for Entity Matching](https://mlsanity.com/paper/2305.03423/) (2023)
- [Noise-Robust De-Duplication at Scale](https://mlsanity.com/paper/2210.04261/) (2022)
- [Contrastive Representation Learning for Cross-Document Coreference Resolution of Events and Entities](https://mlsanity.com/paper/2205.11438/) (2022)
- [Quantifying Character Similarity with Vision Transformers](https://mlsanity.com/paper/2305.14672/) (2023)
- [Linking Representations with Multimodal Contrastive Learning](https://mlsanity.com/paper/2304.03464/) (2023)

## Built on by (most cited first, 4 of 4)

- [Generalizable and Scalable Multistage Biomedical Concept Normalization Leveraging Large Language Models](https://mlsanity.com/paper/2405.15122/) (2024)
- [WAFFLE: Multimodal Floorplan Understanding in the Wild](https://mlsanity.com/paper/2412.00955/) (2024)
- [News Deja Vu: Connecting Past and Present with Semantic Search](https://mlsanity.com/paper/2406.15593/) (2024)
- [Leveraging Language Models for Automated Patient Record Linkage](https://mlsanity.com/paper/2504.15261/) (2025)

## Abstract (arXiv)

Linking information across sources is fundamental to a variety of analyses in social science, business, and government. While large language models (LLMs) offer enormous promise for improving record linkage in noisy datasets, in many domains approximate string matching packages in popular softwares such as R and Stata remain predominant. These packages have clean, simple interfaces and can be easily extended to a diversity of languages. Our open-source package LinkTransformer aims to extend the familiarity and ease-of-use of popular string matching methods to deep learning. It is a general purpose package for record linkage with transformer LLMs that treats record linkage as a text retrieval problem. At its core is an off-the-shelf toolkit for applying transformer models to record linkage with four lines of code. LinkTransformer contains a rich repository of pre-trained transformer semantic similarity models for multiple languages and supports easy integration of any transformer language model from Hugging Face or OpenAI. It supports standard functionality such as blocking and linking on multiple noisy fields. LinkTransformer APIs also perform other common text data processing tasks, e.g., aggregation, noisy de-duplication, and translation-free cross-lingual linkage. Importantly, LinkTransformer also contains comprehensive tools for efficient model tuning, to facilitate different levels of customization when off-the-shelf models do not provide the required accuracy. Finally, to promote reusability, reproducibility, and extensibility, LinkTransformer makes it easy for users to contribute their custom-trained models to its model hub. By combining transformer language models with intuitive APIs that will be familiar to many users of popular string matching packages, LinkTransformer aims to democratize the benefits of LLMs among those who may be less familiar with deep learning frameworks.