Evaluating Transfer Learning Methods on Real-World Data Streams: A Case Study in Financial Fraud Detection

ArXiv ID: 2508.02702 “View on arXiv”

Authors: Ricardo Ribeiro Pereira, Jacopo Bono, Hugo Ferreira, Pedro Ribeiro, Carlos Soares, Pedro Bizarro

Abstract

When the available data for a target domain is limited, transfer learning (TL) methods can be used to develop models on related data-rich domains, before deploying them on the target domain. However, these TL methods are typically designed with specific, static assumptions on the amount of available labeled and unlabeled target data. This is in contrast with many real world applications, where the availability of data and corresponding labels varies over time. Since the evaluation of the TL methods is typically also performed under the same static data availability assumptions, this would lead to unrealistic expectations concerning their performance in real world settings. To support a more realistic evaluation and comparison of TL algorithms and models, we propose a data manipulation framework that (1) simulates varying data availability scenarios over time, (2) creates multiple domains through resampling of a given dataset and (3) introduces inter-domain variability by applying realistic domain transformations, e.g., creating a variety of potentially time-dependent covariate and concept shifts. These capabilities enable simulation of a large number of realistic variants of the experiments, in turn providing more information about the potential behavior of algorithms when deployed in dynamic settings. We demonstrate the usefulness of the proposed framework by performing a case study on a proprietary real-world suite of card payment datasets. Given the confidential nature of the case study, we also illustrate the use of the framework on the publicly available Bank Account Fraud (BAF) dataset. By providing a methodology for evaluating TL methods over time and in realistic data availability scenarios, our framework facilitates understanding of the behavior of models and algorithms. This leads to better decision making when deploying models for new domains in real-world environments.

Keywords: Transfer Learning, Data Availability Simulation, Domain Adaptation, Concept Drift, Covariate Shift, Payment Systems / Financial Fraud

Complexity vs Empirical Score

• Math Complexity: 3.0/10
• Empirical Rigor: 7.0/10
• Quadrant: Street Traders
• Why: The paper introduces a practical framework for simulating dynamic data availability and domain shifts for transfer learning evaluation, which is highly empirical with real-world datasets and implementation details, while the mathematics involved is relatively straightforward, focusing on problem formulation and data manipulation rather than advanced theoretical derivations.

  flowchart TD
    A["Research Goal: Evaluate TL methods<br>under dynamic real-world data streams"] --> B["Core Methodology: Data Manipulation Framework"]

    B --> C["Scenario Simulation<br>Varying data/label availability over time"]
    B --> D["Multi-Domain Creation<br>Resampling to create source/target domains"]
    B --> E["Domain Transformations<br>Apply covariate & concept shifts"]

    C & D & E --> F["Computational Process:<br>Simulate 100s of realistic experiment variants"]

    F --> G["Key Findings:<br>Benchmarks TL performance<br>under dynamic conditions<br>Enables better model deployment decisions"]