Paper: arXiv 2410.12825

Authors: Dwipam Katariya, Juan Manuel Origgi, Yage Wang, Thomas Caputo

Abstract

Users engage with financial services companies through multiple channels, often interacting with mobile applications, web platforms, call centers, and physical locations to service their accounts. The resulting interactions are recorded at heterogeneous temporal resolutions across these domains. This multi-channel data can be combined and encoded to create a comprehensive representation of the customer’s journey for accurate intent prediction. This demands sequential learning solutions. NMT transformers achieve state-of-the-art sequential representation learning by encoding context and decoding for the next best action to represent long-range dependencies. However, three major challenges exist while combining multi-domain sequences within an encoder-decoder transformers architecture for intent prediction applications: a) aligning sequences with different sampling rates b) learning temporal dynamics across multi-variate, multi-domain sequences c) combining dynamic and static sequences. We propose an encoder-decoder transformer model to address these challenges for contextual and sequential intent prediction in financial servicing applications. Our experiments show significant improvement over the existing tabular method.

Complexity vs Empirical Score

  • Math Complexity: 3.5/10
  • Empirical Rigor: 6.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper adapts Transformer architectures (encoder-decoder with ALiBi-style time encodings) to handle multi-domain, heterogeneous temporal financial sequence data at scale. While mathematically straightforward, it demonstrates practical empirical utility on a large real-world industrial dataset (30M+ user sequences), though reproducibility is hindered by proprietary data.

Research Flowchart

  flowchart TD
    A[Research Goal: Predict User Intent Across Multi-Channel Financial Services] --> B[Data Inputs: Multi-Domain Dynamic Sequences & Static Customer Data]
    B --> C[Challenge: Align Heterogeneous Temporal Resolutions & Sampling Rates]
    C --> D[TIMeSynC Model: Encoder-Decoder Transformer with Synchronized Context Encodings]
    D --> E[Computational Process: Sequential Learning & Temporal Dynamics Modeling]
    E --> F[Key Outcome: Significant Improvement in Intent Prediction over Tabular Baselines]

Cited References (47)

  1. Self-Attentive Sequential Recommendation (2018)
  2. Incorporating Time in Sequential Recommendation Models (2023)
  3. Pinnerformer: Sequence modeling for user representation at pinterest (2022)
  4. An intelligent approach to credit card fraud detection using an optimized light gradient boosting machine (2020)
  5. Deep learning enhancing banking services: a hybrid transaction classification and cash flow prediction approach (2022)
  6. DeepTLF: robust deep neural networks for heterogeneous tabular data (2023)
  7. Xgboost: A scalable tree boosting system (2016)
  8. Credit risk assessment based on gradient boosting decision tree (2020)
and 39 more
  1. Credit risk scoring analysis based on machine learning models (2019)
  2. Fata-trans: Field and time-aware transformer for sequential tabular data (2023)
  3. Tabular transformers for modeling multivariate time series (2021)
  4. Top-k off-policy correction for a REINFORCE recommender system (2019)
  5. Recurrent neural networks with top-k gains for session-based recommendations (2018)
  6. GenRec: Generative Personalized Sequential Recommendation (2024) — link
  7. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer (2019)
  8. Transformers4rec: Bridging the gap between nlp and sequential/session-based recommendation (2021)
  9. Transformers with multi-modal features and post-fusion context for e-commerce session-based recommendation (2021) — link
  10. G-TransRec: A Transformer-Based Next-Item Recommendation With Time Prediction (2024)
  11. Attention-based context-aware sequential recommendation model (2020)
  12. Sequential recommendation with graph neural networks (2021)
  13. Behavior sequence transformer for e-commerce recommendation in alibaba (2019)
  14. One transformer for all time series: Representing and training with time-dependent heterogeneous tabular data (2023) — link
  15. Tabnet: Attentive interpretable tabular learning (2021)
  16. Bert: Pre-training of deep bidirectional transformers for language understanding (2018) — link
  17. Vivit: A video vision transformer (2021)
  18. Context-dependent sentiment analysis in user-generated videos (2017)
  19. Memory fusion network for multi-view sequential learning (2018)
  20. Learning factorized multimodal representations (2018) — link
  21. Multimodal transformer for unaligned multimodal language sequences (2019)
  22. Multimodal affective analysis using hierarchical attention strategy with word-level alignment (2018)
  23. Attention is all you need (2017)
  24. Train short, test long: Attention with linear biases enables input length extrapolation (2021) — link
  25. Transact: Transformer-based realtime user action model for recommendation at pinterest (2023)
  26. Lignn: Graph neural networks at linkedin (2024)
  27. adSformers: Personalization from Short-Term Sequences and Diversity of Representations in Etsy Ads (2023) — link
  28. Scaling User Modeling: Large-scale Online User Representations for Ads Personalization in Meta (2024)
  29. Generalized User Representations for Transfer Learning (2024) — link
  30. Augmenting Netflix Search with In-Session Adapted Recommendations (2022)
  31. IntentRec: Predicting User Session Intent with Hierarchical Multi-Task Learning (2024) — link
  32. Exploiting intent evolution in e-commercial query recommendation (2023)
  33. Tablegpt: Towards unifying tables, nature language and commands into one gpt (2023) — link
  34. Tabfact: A large-scale dataset for table-based fact verification (2019) — link
  35. Tabula: Harnessing language models for tabular data synthesis (2023) — link
  36. TaBERT: Pretraining for joint understanding of textual and tabular data (2020) — link
  37. Survey of machine learning algorithms for disease diagnostic (2017)
  38. Contextual and Sequential User Embeddings for Large-Scale Music Recommendation (2020) — link
  39. Revisiting deep learning models for tabular data (2021)

0 of 47 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.