Paper: arXiv 2611.01236

Authors: Nazanin Yousefi, Mohammadreza Ghafouri, Seyedreza Tavakoli, MohammadJavad Gharegozlou, Arman Khaledian, Amir Amini, Eghbal Rahimikia, Gholamali Aminian, Marjan Alirezaie, Ali Habibnia

Abstract

Robust financial-risk inference requires more than mathematical correctness. Models must accurately apply conditional rules, maintain evolving financial states, and make decisions that respect operational constraints. This work introduces RiskReason, a benchmark for verifiable financial-risk reasoning that spans diverse risk-management mechanisms. The benchmark includes Standard tasks, which assess individual financial mechanisms, and Long-horizon tasks, which combine multiple mechanisms through extended dependencies. Each task provides reference outcomes, formal specifications, reasoning annotations, and dependency representations to support systematic analysis of model performance. Eleven model configurations are evaluated using strict single-attempt accuracy, complemented by structural and qualitative failure analyses. The strongest configurations achieve 80.0–90.2% overall accuracy, with several maintaining high performance on composed financial scenarios. However, analysis of failures reveals incorrect risk decisions despite accurate intermediate calculations, inconsistent propagation of financial states, and remediation actions that violate binding constraints. In contrast, some strict evaluation failures result from incomplete reporting rather than incorrect financial conclusions. These findings indicate that aggregate accuracy alone is insufficient to characterize financial-risk reasoning reliability and motivate evaluations that distinguish numerical correctness, state consistency, constraint satisfaction, and decision validity.

Complexity vs Empirical Score

  • Math Complexity: 5.0/10
  • Empirical Rigor: 6.0/10 (capped from 7.0: no out-of-sample test; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces a novel benchmark for financial risk reasoning, emphasizing verifiable outcomes and state consistency beyond mere numerical accuracy. It demonstrates strong empirical rigor through systematic evaluation of multiple models and detailed failure analysis, while also incorporating formal specifications and DAGs for structured reasoning.

Research Flowchart

  flowchart TD
    A[Research Goal: Verifiable Financial Risk Reasoning] --> B[Methodology: Introduce RiskReason Benchmark]
    B --> C{Data/Inputs: Standard & Long-horizon Tasks};
    C --> D[Computational Process: Evaluate 11 Model Configurations];
    D --> E[Outcome 1: Aggregate Accuracy (80.0-90.2%)];
    D --> F[Outcome 2: Failure Analysis (Incorrect Decisions, Inconsistent States, Constraint Violations)];
    E & F --> G[Conclusion: Aggregate Accuracy Insufficient; Need Evaluation of Correctness, Consistency, Satisfaction, Validity];