Advancing Anomaly Detection: Non-Semantic Financial Data Encoding with LLMs

Paper: arXiv 2406.03614

Authors: Alexander Bakumenko, Kateřina Hlaváčková-Schindler, Claudia Plant, Nina C. Hubig

Abstract

Detecting anomalies in general ledger data is of utmost importance to ensure trustworthiness of financial records. Financial audits increasingly rely on machine learning (ML) algorithms to identify irregular or potentially fraudulent journal entries, each characterized by a varying number of transactions. In machine learning, heterogeneity in feature dimensions adds significant complexity to data analysis. In this paper, we introduce a novel approach to anomaly detection in financial data using Large Language Models (LLMs) embeddings. To encode non-semantic categorical data from real-world financial records, we tested 3 pre-trained general purpose sentence-transformer models. For the downstream classification task, we implemented and evaluated 5 optimized ML models including Logistic Regression, Random Forest, Gradient Boosting Machines, Support Vector Machines, and Neural Networks. Our experiments demonstrate that LLMs contribute valuable information to anomaly detection as our models outperform the baselines, in selected settings even by a large margin. The findings further underscore the effectiveness of LLMs in enhancing anomaly detection in financial journal entries, particularly by tackling feature sparsity. We discuss a promising perspective on using LLM embeddings for non-semantic data in the financial context and beyond.

Complexity vs Empirical Score

• Math Complexity: 2.5/10
• Empirical Rigor: 6.5/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper applies sentence-transformer LLM embeddings to non-semantic tabular accounting data to resolve sparsity and dimension heterogeneity issues in anomaly detection. While mathematically simple (focusing on applied deep learning embeddings and standard ML classifiers), it proposes a creative empirical methodology, though data confidentiality constraints reduce reproducibility.

Research Flowchart

  flowchart TD
    A[Research Goal: Detect Financial Anomalies & Handle Feature Sparsity] --> B[Data Input: Real-World General Ledger Journal Entries]
    B --> C[Encoding: 3 Pre-trained Sentence-Transformer LLM Models]
    C --> D[Downstream Classification: 5 ML Models<br>LR, RF, GBM, SVM, NN]
    D --> E[Evaluation: Compare LLM Embeddings vs. Baseline Methods]
    E --> F[Key Outcome: LLM Embeddings Outperform Baselines & Effectively Tackle Sparsity]

Cited References (58)

1. Revolutionary text clustering: Investigating transfer learning capacity of SBERT models through pooling techniques (2024) — link
2. Evaluating Embeddings from Pre-Trained Language Models and Knowledge Graphs for Educational Content Recommendation (2023) — link
3. Data Augmentation for Sample Efficient and Robust Document Ranking (2023) — link
4. Credit Card Analytics: A Review of Fraud Detection and Risk Assessment Techniques (2023) — link
5. Frozen Transformers in Language Models Are Effective Visual Encoder Layers (2023) — link
6. Large Language Models in Finance: A Survey — → 2311.10723
7. Company Similarity Using Large Language Models — → 2308.08031
8. Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs (2023) — link

2 of 58 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.