Paper: arXiv 2611.10972

Authors: Diego Silva Prado, Tiago Amaral Penteado de Jesus, Bruno Caldas de Souza, Cristiano Bevitori Maffia De Oliveira

Abstract

Off-policy evaluation (OPE) estimates the value a candidate decision policy would achieve, using only data logged by the policy already in production. In debt collections, unrecorded factors can influence both actions and outcomes. This paper presents JUBILEE, a benchmark environment built around a debt-collections simulator that generates million-loan panels with competing risks, rare recovery events, and hidden factors driving both outcomes and assignment. The evaluation includes an 80-run comparison of tabular and gradient-boosted estimators under two synthetic outcome mechanisms and a 190-run parameter grid. Independent policy simulations provide reference values with estimated Monte Carlo uncertainty for a synthetic reward score and for simulated cash. With hidden factors influencing action assignment, several tabular methods agree closely while underestimating the field-visit policy’s score, an error that also appears with boosted models. Intervals from tabular doubly robust estimation usually contain the reference without hidden assignment dependence but mostly miss it when that dependence is present. Estimates from randomized decisions do not show the same large error, although their limited precision leaves policy choice uncertain. The decision consequences depend on how policy value is defined. The field-visit policy ranks last under the score and first under simulated cash, which excludes channel costs. Under cash, log-based estimators miss this best candidate in nearly every run with hidden assignment dependence in the paired study. JUBILEE supports reproducible study of these failure modes under specified collections mechanisms.

Complexity vs Empirical Score

  • Math Complexity: 6.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces a novel benchmark environment for off-policy evaluation in a complex domain, demonstrating strong empirical rigor through extensive simulations and comparisons. The methodology is clearly articulated, and the provision of code and data ensures high reproducibility. The mathematical models for the simulator and logging policy are well-defined and contribute to the paper’s robustness.

Research Flowchart

  flowchart TD
    A[Research Goal: Evaluate Off-Policy Evaluation (OPE) Methods in Debt Collections under Hidden Factors] --> B{JUBILEE Benchmark Environment};
    B --> C[Data/Inputs: Debt Collections Simulator (Million-loan panels, competing risks, rare recovery, hidden factors)];
    C --> D{Computational Process: 80-run tabular/gradient-boosted OPE, 190-run parameter grid, Independent policy simulations};
    D --> E[Key Findings: OPE methods (tabular/boosted) underestimate field-visit policy score with hidden assignment dependence; Doubly robust intervals miss reference with hidden dependence; Randomized decisions reduce error but lack precision; Policy ranking depends on value definition (score vs. cash); Log-based estimators miss best policy under cash with hidden assignment dependence.];
    E --> F[Outcome: JUBILEE supports reproducible study of OPE failure modes];