Paper: arXiv 2611.03302

Authors: Denis Burakov

Abstract

A model pretrained to predict a customer’s next event can estimate the probability of future events without training on their labels and provide a reusable representation for supervised downstream banking tasks. We pretrain EventGPT, a hierarchical causal transformer, by next-token maximum likelihood estimation on the public Multimodal Banking Dataset (MBD-mini: 100,224 clients, 38.8 million transactions and 66.3 million geolocation pings). Keeping one state per event, EventGPT trains in about two hours on one L4 GPU at windows up to 8,192 tokens. From its next-event distribution, with no labels, EventGPT scores five targets of the form “the client makes a transaction of a given kind for the first time next month” at ROC-AUC 0.79 to 0.92. Gradient boosting on 114 engineered features needs 10,000 to 300,000 labeled client-months to match it on four targets and stays below it on the fifth. For product prediction, frozen EventGPT embeddings score below the engineered features (mean ROC-AUC 0.787 against 0.808 over five test folds), and a rank-averaged ensemble of EventGPT, profile fields and token counts with the engineered features adds 0.014 to 0.017, of which 0.007 to 0.016 is EventGPT’s own. The code and pretrained weights are released.

Complexity vs Empirical Score

  • Math Complexity: 6.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper presents a novel hierarchical causal transformer for banking event sequences, demonstrating strong empirical performance in zero-shot prediction and offering reusable embeddings. Its high reproducibility with public data and released code/weights significantly bolsters its impact. The mathematical underpinnings of the transformer architecture and temporal embeddings are solid.

Research Flowchart

  flowchart TD
    A[Research Goal: Predict Next Customer Event & Create Reusable Representation] --> B{Methodology: EventGPT Pretraining}
    B --> C[Data: MBD-mini (100k clients, 38.8M transactions, 66.3M pings)]
    C --> D[Computational Process: Hierarchical Causal Transformer, Next-token MLE, 2 hrs on 1 L4 GPU]
    D --> E{Key Finding 1: Unsupervised Next-Event Prediction}
    E --> E1[ROC-AUC 0.79-0.92 for 5 targets (first-time transactions) without labels]
    D --> F{Key Finding 2: Reusable Embeddings for Supervised Tasks}
    F --> F1[Frozen EventGPT embeddings: mean ROC-AUC 0.787 for product prediction]
    F --> F2[Ensemble (EventGPT + engineered features): +0.014 to 0.017 ROC-AUC]