# Enhancing Foundation Models in Transaction Understanding with LLM-based Sentence Embeddings

arXiv: https://arxiv.org/abs/2601.05271 · HTML: https://mlsanity.com/paper/2601.05271/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 1 Dec 2025.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science](https://mlsanity.com/paper/2307.09249/) (2023)
- [LLM2Rec: Large Language Models Are Powerful Embedding Models for Sequential Recommendation](https://mlsanity.com/paper/2506.21579/) (2025)
- [FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data](https://mlsanity.com/paper/2310.13818/) (2023)
- [Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences](https://mlsanity.com/paper/2401.01641/) (2024)
- [TREASURE: The Visa Payment Foundation Model for High-Volume Transaction Understanding](https://mlsanity.com/paper/2511.19693/) (2025)

## Built on by (most cited first, 2 of 2)

- [Understanding LLM Evaluator Behavior: A Structured Multi-Evaluator Framework for Merchant Risk Assessment](https://mlsanity.com/paper/2602.05110/) (2026)
- [Behavior-Grounded Semantic Enrichment for Financial Fraud Modeling and Reasoning](https://mlsanity.com/paper/2609.34211/) (2026)

## Abstract (arXiv)

The ubiquity of payment networks generates vast transactional data encoding rich consumer and merchant behavioral patterns. Recent foundation models for transaction analysis process tabular data sequentially but rely on index-based representations for categorical merchant fields, causing substantial semantic information loss by converting rich textual data into discrete tokens. While Large Language Models (LLMs) can address this limitation through superior semantic understanding, their computational overhead challenges real-time financial deployment. We introduce a hybrid framework that uses LLM-generated embeddings as semantic initializations for lightweight transaction models, balancing interpretability with operational efficiency. Our approach employs multi-source data fusion to enrich merchant categorical fields and a one-word constraint principle for consistent embedding generation across LLM architectures. We systematically address data quality through noise filtering and context-aware enrichment. Experiments on large-scale transaction datasets demonstrate significant performance improvements across multiple transaction understanding tasks.