Could Large Language Models work as Post-hoc Explainability Tools in Credit Risk Models?

Paper: arXiv 2602.18895

Authors: Wenxi Geng, Dingyuan Liu, Liya Li, Yiqing Wang

Abstract

Post-hoc explainability is central to credit risk model governance, yet widely used tools such as coefficient-based attributions and SHapley Additive exPlanations (SHAP) often produce numerical outputs that are difficult to communicate to non-technical stakeholders. This paper investigates whether large language models (LLMs) can serve as post-hoc explainability tools for credit risk predictions through in-context learning, focusing on two roles: translators and autonomous explainers. Using a personal lending dataset from LendingClub, we evaluate three commercial LLMs, including GPT-4-turbo, Claude Sonnet 4, and Gemini-2.0-Flash. Results provide strong evidence for the translator role. In contrast, autonomous explanations show low alignment with model-based attributions. Few-shot prompting improves feature overlap for logistic regression but does not consistently benefit XGBoost, suggesting that LLMs have limited capacity to recover non-linear, interaction-driven reasoning from prompt cues alone. Our findings position LLMs as effective narrative interfaces grounded in auditable model attributions, rather than as substitutes for post-hoc explainers in credit risk model governance. Practitioners should leverage LLMs to bridge the communication gap between complex model outputs and regulatory or business stakeholders, while preserving the rigor and traceability required by credit risk governance frameworks.

Complexity vs Empirical Score

• Math Complexity: 2.5/10
• Empirical Rigor: 6.5/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: This paper addresses a highly practical and novel question regarding the governance of LLMs as translators versus autonomous explainers for ML models in credit risk. While mathematical complexity is minimal, the empirical evaluation using multiple modern LLMs (GPT-4-turbo, Claude Sonnet 4, Gemini-2.0-Flash) on a standard dataset provides robust and actionable insights for model risk management.

Research Flowchart

  flowchart TD
    A["Research Goal:<br>Evaluate LLMs as Post-hoc Explainability Tools in Credit Risk"] --> B["Data & Models:<br>LendingClub Dataset applied to Logistic Regression & XGBoost"]
    
    B --> C["Generate Baseline Attributions:<br>Coefficients (LR) & SHAP Values (XGBoost)"]
    
    C --> D{"Evaluate LLMs in Two Roles<br>(GPT-4-turbo, Claude Sonnet 4, Gemini-2.0-Flash)"}
    
    D -->|"Role 1: Translator<br>(Convert attributions to text)"| E["Outcome: High Alignment<br>Effective narrative interface for stakeholders"]
    
    D -->|"Role 2: Autonomous Explainer<br>(Infer features via Zero/Few-Shot)"| F["Outcome: Low Alignment<br>Fails to recover non-linear, interaction-driven reasoning"]

    E & F --> G["Conclusion:<br>LLMs should translate auditable attributions, not replace post-hoc explainers"]

Cited References (36)

1. FinTech-Issued Personal Loans in the US (2023)
2. Large Language models in critical care medicine: scoping review (2025)
3. AKI-Detector: A Multi-Agent Framework by Integrating Machine Learning and Large Language Models for Early Prediction of Acute Kidney Injury in ICU (2025)
4. Large language model agents can use tools to perform clinical calculations (2025)
5. XGBoost: A Scalable Tree Boosting System (2016)
6. Language models are few-shot learners (2020)
7. Statistical Classification Methods in Consumer Credit Scoring: A Review (1997) — link
8. Benchmarking State-of-the-Art Classification Algorithms for Credit Scoring: An Update of Research (2015) — link

0 of 36 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.