Paper: arXiv 2507.01991

Authors: Muhammad Bilal Zafar

Abstract

The proliferation of artificial intelligence (AI) in financial services has prompted growing demand for tools that can systematically detect AI-related disclosures in corporate filings. While prior approaches often rely on keyword expansion or document-level classification, they fall short in granularity, interpretability, and robustness. This study introduces FinAI-BERT, a domain-adapted transformer-based language model designed to classify AI-related content at the sentence level within financial texts. The model was fine-tuned on a manually curated and balanced dataset of 1,586 sentences drawn from 669 annual reports of U.S. banks (2015 to 2023). FinAI-BERT achieved near-perfect classification performance (accuracy of 99.37 percent, F1 score of 0.993), outperforming traditional baselines such as Logistic Regression, Naive Bayes, Random Forest, and XGBoost. Interpretability was ensured through SHAP-based token attribution, while bias analysis and robustness checks confirmed the model’s stability across sentence lengths, adversarial inputs, and temporal samples. Theoretically, the study advances financial NLP by operationalizing fine-grained, theme-specific classification using transformer architectures. Practically, it offers a scalable, transparent solution for analysts, regulators, and scholars seeking to monitor the diffusion and framing of AI across financial institutions.

Complexity vs Empirical Score

  • Math Complexity: 3.5/10
  • Empirical Rigor: 8.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper employs standard deep learning fine-tuning and statistical metrics (F1, accuracy) with SHAP for interpretability, resulting in moderate mathematical complexity. It demonstrates high empirical rigor through a detailed, manually curated dataset, explicit implementation instructions (Hugging Face), and robustness checks.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Detect AI Disclosures<br>in Financial Reports"] --> B
  subgraph B ["Methodology & Data"]
      direction LR
      B1["Dataset<br>1,586 Sentences<br>U.S. Banks (2015-2023)"] --> B2["FinAI-BERT<br>Domain-Adapted Transformer"]
  end
  B --> C["Computational Process<br>Fine-tuning & Classification"]
  C --> D{"Evaluation & Analysis"}
  D --> E["Outcomes<br>99.37% Accuracy / F1 0.993<br>SHAP Interpretability"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes FinAI-BERT, a domain-adapted transformer-based language model for sentence‑level classification of AI disclosures in financial texts.

Source sentence
This study introduces FinAI‑BERT, a domain‑adapted transformer‑based language model designed to classify AI‑related content at the sentence level within financial texts.

Assumes

No explicit assumptions found in the text.

Data

  • FinAI‑BERT sentence dataset text-news unknown
    2015–2023 · 669 U.S. bank annual reports
    Source sentence
    The model was fine‑tuned on a manually curated and balanced dataset of 1,586 sentences drawn from 669 annual reports of U.S. banks (2015–2023).

Costs

Not applicable The paper reports no trading or execution result.

Finds 2

  • FinAI‑BERT achieved near‑perfect classification performance with an accuracy of 99.37% and a macro F1‑score of 0.993 on the held‑out test set. performance
    Source sentence
    On the test set, FinAI‑BERT achieved an accuracy of 99.37% and a macro F1‑score of 0.993, indicating highly reliable classification performance across both classes.
  • The model correctly classified all AI and non‑AI sentences, yielding a perfect confusion matrix with no observable misclassifications. performance
    Source sentence
    The model correctly classified 100% of Non‑AI sentences (n = 231) and 100% of AI‑related sentences (n = 159), yielding a perfect confusion matrix and no observable misclassifications

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 19 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

https://huggingface.co/bilalzafar/BankAI-BERT — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.