Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs

Paper: arXiv 2407.17624

Authors: Felix Drinkall, Janet B. Pierrehumbert, Stefan Zohren

Abstract

Large Language Models (LLMs) have been shown to perform well for many downstream tasks. Transfer learning can enable LLMs to acquire skills that were not targeted during pre-training. In financial contexts, LLMs can sometimes beat well-established benchmarks. This paper investigates how well LLMs perform in the task of forecasting corporate credit ratings. We show that while LLMs are very good at encoding textual information, traditional methods are still very competitive when it comes to encoding numeric and multimodal data. For our task, current LLMs perform worse than a more traditional XGBoost architecture that combines fundamental and macroeconomic data with high-density text-based embedding features.

Complexity vs Empirical Score

• Math Complexity: 3.0/10
• Empirical Rigor: 8.0/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: This paper presents a rigorous empirical study comparing generative LLMs with traditional boosted trees (XGBoost) for credit rating forecasting using 23 years of corporate data. It excels in empirical evaluation, clarity, and open-science commitment by making code and dataset reconstruction tools available, though it relies on standard machine learning algorithms rather than complex mathematical derivations.

Research Flowchart

  flowchart TD
    A[Research Goal: Forecast Corporate Credit Ratings LLMs vs. Traditional] --> B[Data Integration]
    
    B --> C1[Fundamental & Macroeconomic Numeric Data]
    B --> C2[High-Density Text-Based Embeddings]
    
    C1 --> D[XGBoost Model Architecture]
    C2 --> D
    C2 --> E[Generative LLMs Transfer Learning]
    
    D --> F[Outcome: XGBoost Outperforms LLMs on Multimodal Data]
    E --> F