Paper: arXiv 2610.26797

Authors: Agus Sudjianto, Denis Burakov

Abstract

Candidate-scoring decision models return a probability distribution over a presented set of answers. Public documentation of one such system, Jev, and of an open counterpart, JevAny, describes three training ingredients that determine what these probabilities estimate: soft targets from larger models, a frozen pretrained backbone and temperature calibration. We analyze each ingredient in Gaussian surrogates where the finite-sample risk can be computed exactly. A student trained on teacher targets inherits the teacher’s bias, which more student data does not remove and agreement with the teacher does not reveal. A fixed representation lowers the risk only below an explicit sample-size threshold. In a probit model the exact recalibration map is a temperature scaling, which removes all excess risk only when the teacher’s coefficient has the correct direction and otherwise leaves a residual that calibration diagnostics do not detect. Under Luce’s choice axiom, candidate scores are coherent across refinements of an event if and only if they aggregate by log-sum-exp. On 231,521 credit card complaints from the Consumer Financial Protection Bureau, where the decision is whether a complaint ends in monetary relief, a student trained on a logistic teacher’s probabilities ends at the teacher’s decision cost while a student trained on outcomes goes 8% below it, a 30B model read zero-shot remains worse than the teacher after recalibration, and that model departs from the coherence condition by 17.5 decibans. Pretrained representations reduce the labeled data required and calibration corrects the scale of the probabilities, but neither corrects a biased teacher, so the probabilities have to be checked against realized outcomes.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: This paper presents a highly mathematical analysis of candidate-scoring models, deriving exact risk calculations under Gaussian surrogates. It then rigorously tests these theoretical findings with both Monte Carlo simulations and a real-world credit card complaints dataset. The novel approach of dissecting teacher supervision, fixed representations, and calibration with exact calculations provides significant new insights into model behavior.

Research Flowchart

  flowchart TD
    A[Research Goal: Exact Risk Calculations for Candidate-Scoring Decision Models] --> B{Key Methodology: Gaussian Surrogates Analysis};
    B --> C{Inputs: Jev/JevAny Documentation, Credit Card Complaint Data (231,521)};
    C --> D{Computational Processes: Teacher-Student Learning, Fixed Representation Analysis, Temperature Calibration, Probit Model Analysis, Log-Sum-Exp Aggregation};
    D --> E[Key Findings/Outcomes];
    E --> F{Teacher's Bias Inherited by Student, Fixed Representation Benefit is Sample-Size Dependent};
    E --> G{Calibration Removes Excess Risk Only Under Specific Teacher Conditions, Residual Bias Undetected};
    E --> H{Student Trained on Teacher Probabilities = Teacher's Decision Cost; Student on Outcomes < Teacher's Cost (8% below)};
    E --> I{Large Language Model (30B) Worse Than Teacher, Departs from Coherence Condition (17.5 decibans)};
    E --> J{Conclusion: Pretrained Reps & Calibration Don't Correct Biased Teacher; Probabilities Must Be Checked Against Realized Outcomes};