Paper: arXiv 2508.02702

Authors: Ricardo Ribeiro Pereira, Jacopo Bono, Hugo Ferreira, Pedro Ribeiro, Carlos Soares, Pedro Bizarro

Abstract

When the available data for a target domain is limited, transfer learning (TL) methods can be used to develop models on related data-rich domains, before deploying them on the target domain. However, these TL methods are typically designed with specific, static assumptions on the amount of available labeled and unlabeled target data. This is in contrast with many real world applications, where the availability of data and corresponding labels varies over time. Since the evaluation of the TL methods is typically also performed under the same static data availability assumptions, this would lead to unrealistic expectations concerning their performance in real world settings. To support a more realistic evaluation and comparison of TL algorithms and models, we propose a data manipulation framework that (1) simulates varying data availability scenarios over time, (2) creates multiple domains through resampling of a given dataset and (3) introduces inter-domain variability by applying realistic domain transformations, e.g., creating a variety of potentially time-dependent covariate and concept shifts. These capabilities enable simulation of a large number of realistic variants of the experiments, in turn providing more information about the potential behavior of algorithms when deployed in dynamic settings. We demonstrate the usefulness of the proposed framework by performing a case study on a proprietary real-world suite of card payment datasets. Given the confidential nature of the case study, we also illustrate the use of the framework on the publicly available Bank Account Fraud (BAF) dataset. By providing a methodology for evaluating TL methods over time and in realistic data availability scenarios, our framework facilitates understanding of the behavior of models and algorithms. This leads to better decision making when deploying models for new domains in real-world environments.

Complexity vs Empirical Score

  • Math Complexity: 3.0/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper introduces a practical framework for simulating dynamic data availability and domain shifts for transfer learning evaluation, which is highly empirical with real-world datasets and implementation details, while the mathematics involved is relatively straightforward, focusing on problem formulation and data manipulation rather than advanced theoretical derivations.

Research Flowchart

  flowchart TD
  A["Research Goal: Evaluate TL methods<br>under dynamic real-world data streams"] --> B["Core Methodology: Data Manipulation Framework"]

  B --> C["Scenario Simulation<br>Varying data/label availability over time"]
  B --> D["Multi-Domain Creation<br>Resampling to create source/target domains"]
  B --> E["Domain Transformations<br>Apply covariate & concept shifts"]

  C & D & E --> F["Computational Process:<br>Simulate 100s of realistic experiment variants"]

  F --> G["Key Findings:<br>Benchmarks TL performance<br>under dynamic conditions<br>Enables better model deployment decisions"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes an evaluation framework that simulates dynamic data availability, creates multiple domains via resampling, and applies realistic transformations for transfer learning assessment.

Source sentence
To address this challenge, we propose an evaluation framework that simulates the dynamic nature of real-world data streams. Our framework provides three key capabilities: (1) creating multiple domains from a given dataset through re‑sampling, enabling systematic TL evaluation even when few datasets are available; (2) a

Assumes 5

  • Labels become available after a delay (ty_i ≥ tx_i). data
    Source sentence
    ... tx i is the timestamp when xi is collected, and ty i ≥tx i is the timestamp when yi becomes available.
  • A large volume of labeled data is available from the source domains. data
    Source sentence
    that a large volume of labeled data is available from the source domains.
  • The target domain initially has no data and then data/labels arrive gradually over time. data
    Source sentence
    at some time ta, the target domain DT is introduced, initially without any data (DL T (ta) = DU T (ta) = ∅), and target domain data and labels begin to be collected from that point on.
  • Multiple domains can be generated from a single dataset via resampling. model
    Source sentence
    creating multiple domains from a given dataset through re‑sampling, enabling systematic TL evaluation even when few datasets are available
Show 1 more
  • Realistic domain shifts can be simulated by applying controlled transformations. model
    Source sentence
    applying transformations to the data, hence reproducing realistic data shifts over time and across domains

Data

  • proprietary card payment dataset alternative proprietary
    Source sentence
    We perform a case study using our framework on a suite of proprietary real‑world datasets containing payment events from multiple financial institutions.
  • Bank Account Fraud (BAF) dataset alternative public
    Source sentence
    Given the confidential nature of the case study, we also illustrate the use of the framework on the publicly available Bank Account Fraud (BAF) dataset.

Costs

Not applicable The paper reports no trading or execution result.

Finds 4

  • MTAE, DANN, BL‑A and BL‑S show similar recall levels and maintain stable performance over time, indicating limited benefit from additional target data. performance
    Source sentence
    – MTAE, DANN, BL-A and BL-S show similar levels of recall, maintaining a stable distribution of predictive performance over time. This suggests that there is a limited benefit from the additional target domain data and labels.
  • MAN and BL‑T start worse but improve steadily, gaining about 2 percentage points of recall per model update and eventually match the performance of the other methods. performance
    Source sentence
    – MAN and BL-T begin to perform significantly worse than the previous group of methods, but they improve steadily over time (gaining on average 2 percentage points of recall per model update) and eventually reaching the same level of performance.
  • MME maintains relatively stable performance but is consistently outperformed by the other methods. performance
    Source sentence
    – MME maintains a relatively stable performance throughout, but is consistently surpassed by the other methods.
  • Methods that do not use any target data (MTAE, BL‑S) perform similarly to BL‑T, suggesting source and target domains are relatively similar and knowledge sharing is feasible. predictive-accuracy
    Source sentence
    the performance of methods such as MTAE and BL‑S, which do not use any target data during training, is similar to the performance of BL‑T, which follows the traditional ML approach of only using in‑domain data to train. This suggests that the source and target domains in the BAF dataset are relatively similar, which me

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 33 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

https://github.com/feedzai/tred — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.