Honest Small Decision Models for Finance: Scale, Specialization Collapse, and Mixed-Corpus Remediation

Paper: arXiv 2611.03755

Authors: Ajitabh Kumar

Abstract

Financial workflows need fast, calibrated, auditable decisions, not just generated text. We train System-1 decision models, small language models fine-tuned to answer typed questions (choice/score/noul) with a probability in a single forward pass, across two model families (K2-Horizon and Qwen3.5/3.8) and nine rungs from 0.8B to 32B parameters, using one frozen LoRA recipe on a public decision benchmark and two new finance corpora: a programmatic-gold policy corpus (rule-reading with exact labels) and a reality-labeled outcome corpus (19,640 realized forward-return questions in walk-forward splits, plus a sealed 2026 test of 1,360 questions). Narrow fine-tuning costs general decision competence 9-45 percentage points, a data-mixture effect that shrinks with model scale and is associated with narrow-gradient volume. On reality-labeled questions, every trained rung from 0.8B to 32B lands at the horizon-conditioned prior baseline (accuracy 0.37-0.38, no detectable conditional edge under our controls); the models respond with flat, well-calibrated probabilities that track per-horizon base rates: naturally honest uncertainty, with a mid-scale calibration sweet spot (ECE 0.026 at 2B, degrading to 0.043-0.044 at 9-32B under prior drift). A mixed-corpus remediation carries all competencies in one adapter at modest cost: it matches or slightly exceeds the generalist specialist on the public benchmark while holding the policy ceiling and near-specialist outcome calibration, yielding a deployable artifact with per-domain temperature regimes and a band-dispatch escalation analysis: sharp, dispatchable bands where rules determine answers; flat, review-routing bands where reality does not.

Complexity vs Empirical Score

• Math Complexity: 6.5/10
• Empirical Rigor: 8.0/10
• Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper demonstrates strong empirical rigor through extensive model training, robust evaluation protocols, and the creation of new, carefully constructed finance corpora. Its novelty lies in applying and characterizing small decision models for financial workflows, particularly addressing specialization collapse and honest uncertainty. The methodology is well-detailed, supporting high reproducibility.

Research Flowchart

  flowchart TD
    A[Research Goal: Develop Honest, Small Decision Models for Finance] --> B{Key Methodologies: Fine-tuning SLMs for Decision Tasks};
    B --> C{Data/Inputs: Public Benchmark, Programmatic-Gold Policy Corpus, Reality-Labeled Outcome Corpus};
    C --> D[Computational Processes: Train K2-Horizon & Qwen (0.8B-32B) models with LoRA, Evaluate Accuracy & Calibration];
    D --> E{Key Finding 1: Narrow fine-tuning reduces general competence (9-45pp), effect shrinks with scale};
    E --> F{Key Finding 2: On reality-labeled tasks, models revert to horizon-conditioned prior baseline (accuracy 0.37-0.38), exhibit honest uncertainty & good calibration (ECE 0.026 at 2B)};
    F --> G[Key Finding 3: Mixed-corpus remediation preserves competencies, yields deployable artifact with per-domain temperature & band-dispatch analysis];