Paper: arXiv 2604.08649

Authors: Maxim Ostroukhov, Ruslan Mikhailov, Vladimir Iashin, Artem Sokolov, Andrei Akshonov, Vitaly Protasov, Dmitrii Beloborodov, Vince Mullin, Roman Yokunda Enzmann, Georgios Kolovos, Jason Renders, Pavel Nesterov, Anton Repushko

Abstract

Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for multi-source banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.

Complexity vs Empirical Score

  • Math Complexity: 4.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: PRAGMA introduces an impactful foundation model architecture for multi-source banking event sequences, demonstrating strong empirical utility across tasks like fraud detection and credit scoring. While relying on standard self-supervised Transformer principles rather than deep theoretical mathematics, its extensive evaluation on real-world transactional data highlights strong practical applicability.

Research Flowchart

  flowchart TD
    A[Goal: General-Purpose Representation Model for Multi-Source Banking Events] --> B[(Input: Large-Scale Heterogeneous Banking Event Corpus)]
    B --> C[Method: Pre-train Transformer with Self-Supervised Masked Modeling]
    C --> D[Generate Rich Financial Event Embeddings]
    D --> E[Downstream Tasks: Credit Scoring, Fraud Detection, LTV Prediction]
    E --> F[Evaluation: Linear Probe vs. Lightweight Fine-Tuning]
    F --> G[Outcome: Superior Zero-Shot & Fine-Tuned Performance Across Domains]

Cited References (62)

  1. Attention is all you need (2017)
  2. Decoupled Weight Decay Regularization (2019)
  3. Adam: A Method for Stochastic Optimization (2015)
  4. Bert: Pre-training of deep bidirectional transformers for language understanding (2019)
  5. Language models are few-shot learners (2020)
  6. An image is worth 16x16 words: Transformers for image recognition at scale (2021)
  7. Exploring the limits of transfer learning with a unified text-to-text transformer (2020)
  8. FlashAttention : Fast and memory-efficient exact attention with io-awareness (2022)
and 54 more
  1. Switch Transformers : Scaling to trillion parameter models with simple and efficient sparsity (2022)
  2. Mixtral of experts (2024) — link
  3. GPT-4o system card (2024) — link
  4. Gemini 1.5 : Unlocking multimodal understanding across millions of tokens of context (2024) — link
  5. On the opportunities and risks of foundation models (2021)
  6. RoBERTa : A robustly optimized bert pretraining approach (2019) — link
  7. Masked autoencoders are scalable vision learners (2022)
  8. Data2vec: A general framework for self-supervised learning in speech, vision and language (2022)
  9. Self-supervised learning from images with a joint-embedding predictive architecture (2023)
  10. TabTransformer : Tabular data modeling using contextual embeddings (2020) — link
  11. Revisiting deep learning models for tabular data (2021)
  12. TabPFN : A transformer that solves small tabular classification problems in a second (2023)
  13. Accurate predictions on small data with a tabular foundation model (2025)
  14. TabPFN-2.5 : Advancing the state of the art in tabular foundation models (2025) — link
  15. Trompt : Towards a better deep neural network for tabular data (2023)
  16. Self-attentive sequential recommendation (2018)
  17. BERT4Rec : Sequential recommendation with bidirectional encoder representations from transformer (2019)
  18. Recommendation as language processing ( RLP ): A unified pretrain, personalized prompt & predict paradigm ( P5 ) (2022)
  19. TALLRec : An effective and efficient tuning framework to align large language model with recommendation (2023)
  20. Chat-REC : Towards interactive and explainable LLM s-augmented recommender system (2023) — link
  21. FinBERT : A pretrained language model for financial communications (2020) — link
  22. BloombergGPT : A large language model for finance (2023) — link
  23. FinGPT : Open-source financial large language models (2023)
  24. LoRA : Low-rank adaptation of Large Language Models (2022)
  25. Time-LLM : Time series forecasting by reprogramming large language models (2024)
  26. Chronos : Learning the language of time series (2024)
  27. SAINT : Improved neural networks for tabular data via row attention and contrastive pre-training (2021) — link
  28. DCN V2 : Improved Deep & Cross Network and practical lessons for web-scale learning to rank systems (2021)
  29. Mitra : Mixed Synthetic Priors for Enhancing Tabular Foundation Models (2025)
  30. BEiT : BERT pre-training of image transformers (2021) — link
  31. NV-Retriever: Improving text embedding models with effective hard-negative mining (2024) — link
  32. On the limited memory BFGS method for large scale optimization (1989)
  33. Optuna: A next-generation hyperparameter optimization framework (2019)
  34. QLoRA : Efficient finetuning of quantized llms (2023)
  35. RoFormer : Enhanced transformer with rotary position embedding (2024)
  36. Segment anything (2023)
  37. Emerging properties in self-supervised vision transformers (2021)
  38. LightGBM : A highly efficient gradient boosting decision tree (2017)
  39. Your spending needs attention: Modeling financial habits with transformers (2025) — link
  40. TransactionGPT (2025) — link
  41. The self-normalized estimator for counterfactual learning (2015)
  42. Muon is scalable for LLM training (2025) — link
  43. Neural machine translation of rare words with subword units (2016)
  44. Gaussian error linear units (gelus) (2016) — link
  45. On layer normalization in the transformer architecture (2020)
  46. Dropout: a simple way to prevent neural networks from overfitting (2014)
  47. Muon: An optimizer for hidden layers in neural networks (2024) — link
  48. Rethinking the inception architecture for computer vision (2016)
  49. Metalearners for estimating heterogeneous treatment effects using machine learning (2019)
  50. Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations (2024) — link
  51. TransAct V2 : Lifelong user action sequence modeling on Pinterest recommendation (2025) — link
  52. Scaling recommender Transformers to one billion parameters (2025) — link
  53. TransAct : Transformer-based realtime user action model for recommendation at Pinterest (2023)
  54. On embeddings for numerical features in tabular deep learning (2022)

0 of 62 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.