Paper: arXiv 2505.24650

Authors: Hariom Tatsat, Ariye Shater

Abstract

Large Language Models (LLMs) exhibit remarkable capabilities across a spectrum of tasks in financial services, including report generation, chatbots, sentiment analysis, regulatory compliance, investment advisory, financial knowledge retrieval, and summarization. However, their intrinsic complexity and lack of transparency pose significant challenges, especially in the highly regulated financial sector, where interpretability, fairness, and accountability are critical. As far as we are aware, this paper presents the first application in the finance domain of understanding and utilizing the inner workings of LLMs through mechanistic interpretability, addressing the pressing need for transparency and control in AI systems. Mechanistic interpretability is the most intuitive and transparent way to understand LLM behavior by reverse-engineering their internal workings. By dissecting the activations and circuits within these models, it provides insights into how specific features or components influence predictions - making it possible not only to observe but also to modify model behavior. In this paper, we explore the theoretical aspects of mechanistic interpretability and demonstrate its practical relevance through a range of financial use cases and experiments, including applications in trading strategies, sentiment analysis, bias, and hallucination detection. While not yet widely adopted, mechanistic interpretability is expected to become increasingly vital as adoption of LLMs increases. Advanced interpretability tools can ensure AI systems remain ethical, transparent, and aligned with evolving financial regulations. In this paper, we have put special emphasis on how these techniques can help unlock interpretability requirements for regulatory and compliance purposes - addressing both current needs and anticipating future expectations from financial regulators globally.

Complexity vs Empirical Score

  • Math Complexity: 7.0/10
  • Empirical Rigor: 4.0/10
  • Quadrant: Lab Rats — high math complexity, limited empirical testing

Why this score: The paper introduces advanced machine learning concepts like mechanistic interpretability, sparse autoencoders, and circuit analysis, which involve dense mathematical foundations. However, it focuses on theoretical exploration and conceptual financial use cases (e.g., sentiment analysis, bias detection) without presenting code, specific backtests, datasets, or statistical validation metrics, indicating a lack of empirical implementation depth.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Application of Mechanistic<br>Interpretability in Finance"] --> B["Methodology<br>Theoretical Framework + Experiments"]
  B --> C["Computational Process<br>Reverse-engineering LLM<br>Activations & Circuits"]
  C --> D{"Key Financial Use Cases"}
  D --> E["Trading Strategy Analysis"]
  D --> F["Sentiment & Bias Detection"]
  D --> G["Hallucination Detection"]
  E & F & G --> H["Key Outcomes<br>Enhanced Transparency &<br>Regulatory Compliance"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes applying mechanistic interpretability techniques—such as logit lens, sparse autoencoders, activation patching, and feature steering—to understand and modify LLM behavior for financial tasks.

Source sentence
As far as we are aware, this paper presents the first application in the finance domain of understanding and utilizing the inner workings of LLMs through mechanistic interpretability, addressing the pressing need for transparency and control in AI systems.

Assumes

No explicit assumptions found in the text.

Data

  • SEC filings and financial documents vector database · SEC text-news unknown
    Source sentence
    For this experiment, a vector database containing processed SEC filings and financial documents was prepared; embeddings of document chunks were stored to allow efficient semantic retrieval when needed.
  • Labeled dataset of financial news headlines with price movements text-news unknown
    Source sentence
    Using activations from the SAE of the Gemma-2B model and a labeled dataset of financial news headlines with corresponding price movements, we performed feature discovery to identify the most relevant features from the dataset.

Costs

Not applicable The paper reports no trading or execution result.

Finds 5

  • Mechanistic interpretability tools can isolate crucial risk or sentiment indicators and enable dynamic model adjustments, reducing the need for costly fine‑tuning. other
    Source sentence
    Our experiments showed that these interpretability tools not only help isolate crucial risk or sentiment indicators but also allow dynamic model adjustments—cutting down on costly fine-tuning and enabling more precise responses in high-stakes scenarios.
  • Feature steering shifts model sentiment toward credit‑risk factors, improving classification balance without extensive prompt engineering. performance
    Source sentence
    With feature steering applied at a high magnitude to the feature "concepts related to credit ratings and financial recovery", the sentiment shifts to 4, highlighting both the negative (lower credit score) and positive (secured financing & debt reduction) aspects, ultimately improving the overall outlook. This change occurs because steering shifts the model’s attention toward financially stabilizing factors, ensuring
  • Sparse autoencoder feature activations combined with a decision‑tree classifier achieve interpretable short‑term stock price movement predictions with measurable F1 performance. predictive-accuracy
    Source sentence
    Once the key features were identified, we trained a decision tree classifier to find the optimal combination of features and evaluated its performance using the F1 score. The decision tree classifier, trained on this data, is visualized to show how different features contribute to predicting stock price movements (up or down).
  • Activation‑based feature magnitude can be used for bias detection in loan approval and for hallucination reduction via retrieval‑augmented generation. other
    Source sentence
    6.2.2 Bias Detection and Reduction In this experiment, we address bias reduction in the loan approval process by analyzing feature activations in sparse autoencoders (SAE). 6.2.3 Hallucination Reduction In this experiment, we aimed to enhance the reliability of the language model’s output by mitigating potential hallucinations, specifically in the context of financial information.
  • Logit Lens visualizations reveal how probabilities shift across layers, clarifying which model parts influence specific financial predictions. theoretical
    Source sentence
    5.1 Logit Lens In this experiment, Logit Lens is employed to track how probabilities shift across layers, allowing for a clearer understanding of which parts of the model influence specific financial predictions. First, we used the Logit Lens to generate a heatmap as shown in Figure 5 illustrating the model’s predicted token probabilities at each layer for a given financial prompt.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 21 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 8 papers in the archive (8 by independent authors): Alpha-R1: Alpha Screening with LLM Reasoning via … (discusses its findings); RealFin: How Well Do LLMs Reason About Finance When Users … (discusses its findings)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (57)

  1. Mechanistic Interpretability for AI Safety: A Review (2024) — link
  2. Explainable Artificial Intelligence (XAI) in Finance: A Systematic Literature Review (2024) — link
  3. Mechanistic Interpretability of Large Language Models with Applications to Financial Services Industry (2024) — link
  4. A Survey of Methods for Explaining Black Box Models (2018) — link
  5. Explainable AI: A Review of Machine Learning Interpretability Methods (2020) — link
  6. Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks (2023) — link
  7. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018) — link
  8. Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities, and Challenges toward Responsible AI (2020) — link
and 49 more
  1. A Review of Explainable AI: From Statistical Interpretation to Ethical AI Systems (2023) — link
  2. A Study on the Role of Explainability in Machine Learning Models (2023) — link
  3. Interpretable Deep Learning for Cyber-Physical Systems: A Model-Agnostic Perspective (2021) — link
  4. Regulatory Framework for AI in Europe (2020) — link
  5. Balancing Performance and Explainability in AI Systems (2024) — link
  6. A Unified Approach to Interpreting Model Predictions (2017) — link
  7. Why Should I Trust You? Explaining the Predictions of Any Classifier (2016) — link
  8. Axiomatic Attribution for Deep Networks (2017) — link
  9. Knowledge Distillation: A Survey (2020) — link
  10. Grad-CAM: Visual Explanations from Deep Networks (2016) — link
  11. Probing Classifiers: Promises, Shortcomings, and Advances (2021) — link
  12. Investigating BERT’s Sensitivity to Lexical Semantic Relations (2020) — link
  13. Visualizing the Feature Importance for Black Box Models (2018) — link
  14. The Logit Lens: A Window into Transformer Predictions (2020) — link
  15. Future Lens: Probing Token Representations for Predictive Information (2023) — link
  16. Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1? (1997) — link
  17. Gated Sparse Autoencoders for Improved Feature Learning (2024) — link
  18. Transcoders for Sparse Feature Discovery in Neural Networks (2024) — link
  19. Toy Models of Superposition (2022) — link
  20. Towards Monosemanticity: Sparse Autoencoders for Feature Disentanglement (2023) — link
  21. Mechanistic Interpretability of Indirect Object Identification in Transformers (2023) — link
  22. A Mechanistic Interpretability Analysis of Superposition (2022) — link
  23. In-Context Learning and Induction Heads in Transformers (2022)
  24. Ensuring AI Aligns with Human Values (2023)
  25. Unsupervised Sentiment Neuron (2017) — link
  26. Monosemantic Features and Dictionary Learning (2023) — link
  27. Causality: Models, Reasoning, and Inference (2009)
  28. Understanding Intermediate Layers Using Linear Classifier Probes (2016) — link
  29. Visualizing and Understanding Convolutional Networks (2014) — link
  30. Attention Lens: Exploring Attention Maps in Transformers (2023) — link
  31. Discovering Latent Knowledge in Language Models Without Supervision (2022) — link
  32. Analysis of Activation Patching in Neural Networks (2023) — link
  33. Activation Patching for Explainability (2023) — link
  34. Understanding Transformer Circuits via Resample Ablation (2023)
  35. Gemma 2: Improving Open Language Models at a Practical Size (2024) — link
  36. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024) — link
  37. BERTopic: Neural Topic Modeling with a Class‑Based TF‑IDF Procedure (2022) — link
  38. A Review of Recent Studies on the Development of Smart Cities (2016) — link
  39. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning (2024)
  40. Neuronpedia: A Platform for Mechanistic Interpretability Research (2024)
  41. Training-free Deep Concept Injection Enables Language Models for Video Question Answering (2024) — link
  42. Neural Causal Abstractions (2024) — link
  43. Sparse Autoencoder Steering for Disentangled Representations (2024) — link
  44. SelfIE: Self-Interpretation of Large Language Model Embeddings (2024) — link
  45. A Mathematical Framework for Transformer Circuits (2021) — link
  46. Progress Measures for Grokking via Mechanistic Interpretability (2023) — link
  47. BERTopic: Neural topic modeling with a class-based TF-IDF procedure (2022) — link
  48. Gemma 2: Improving Open Language Models at a Practical Size (2024) — link
  49. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (2024) — link

0 of 57 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.