# Advancing Anomaly Detection: Non-Semantic Financial Data Encoding with LLMs

arXiv: https://arxiv.org/abs/2406.03614 · HTML: https://mlsanity.com/paper/2406.03614/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 58 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs](https://mlsanity.com/paper/2307.03393/) (2023)
- [Large Language Models in Finance: A Survey](https://mlsanity.com/paper/2311.10723/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [AI in Finance: Challenges, Techniques and Opportunities](https://mlsanity.com/paper/2107.09051/) (2021)
- [Frozen Transformers in Language Models Are Effective Visual Encoder Layers](https://mlsanity.com/paper/2310.12973/) (2023)
- [PatentSBERTa: A Deep NLP based Hybrid Model for Patent Distance and Classification using Augmented SBERT](https://mlsanity.com/paper/2103.11933/) (2021)
- [Leveraging Pre-trained BERT for Audio Captioning](https://mlsanity.com/paper/2203.02838/) (2022)
- [Detection of Accounting Anomalies in the Latent Space using Adversarial Autoencoder Neural Networks](https://mlsanity.com/paper/1908.00734/) (2019)
- [Data Augmentation for Sample Efficient and Robust Document Ranking](https://mlsanity.com/paper/2311.15426/) (2023)

## Built on by (most cited first, 3 of 3)

- [Advanced Real-Time Fraud Detection Using RAG-Based LLMs](https://mlsanity.com/paper/2501.15290/) (2025)
- [Enhancing Trust and Safety in Digital Payments: An LLM-Powered Approach](https://mlsanity.com/paper/2410.19845/) (2024)
- [Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attribution](https://mlsanity.com/paper/2511.15847/) (2025)

## Abstract (arXiv)

Detecting anomalies in general ledger data is of utmost importance to ensure trustworthiness of financial records. Financial audits increasingly rely on machine learning (ML) algorithms to identify irregular or potentially fraudulent journal entries, each characterized by a varying number of transactions. In machine learning, heterogeneity in feature dimensions adds significant complexity to data analysis. In this paper, we introduce a novel approach to anomaly detection in financial data using Large Language Models (LLMs) embeddings. To encode non-semantic categorical data from real-world financial records, we tested 3 pre-trained general purpose sentence-transformer models. For the downstream classification task, we implemented and evaluated 5 optimized ML models including Logistic Regression, Random Forest, Gradient Boosting Machines, Support Vector Machines, and Neural Networks. Our experiments demonstrate that LLMs contribute valuable information to anomaly detection as our models outperform the baselines, in selected settings even by a large margin. The findings further underscore the effectiveness of LLMs in enhancing anomaly detection in financial journal entries, particularly by tackling feature sparsity. We discuss a promising perspective on using LLM embeddings for non-semantic data in the financial context and beyond.