{"paper_id":"2505.24650","title":"Beyond the Black Box: Interpretability of LLMs in Finance","authors":["Hariom Tatsat","Ariye Shater"],"paper_date":"2025-05-14","paper_type":"methodological","paper_url":"https://arxiv.org/pdf/2505.24650v1","page_url":"https://thequant.space/flowcharts/beyond-the-black-box-interpretability-of-llms-in-finance/","scores":{"empirical_rigor":4,"math_complexity":7,"quadrant":"Lab Rats"},"proposes":{"status":"available","statement":"The paper proposes applying mechanistic interpretability techniques—such as logit lens, sparse autoencoders, activation patching, and feature steering—to understand and modify LLM behavior for financial tasks.","quote":"As far as we are aware, this paper presents the first application in the finance domain of understanding and utilizing the inner workings of LLMs through mechanistic interpretability, addressing the pressing need for transparency and control in AI systems.","summary":"Large Language Models (LLMs) exhibit remarkable capabilities across a spectrum of tasks in financial services, including report generation, chatbots, sentiment analysis, regulatory compliance, investment advisory, financial knowledge retrieval, and summarization. However, their intrinsic complexity…","strategy":null},"assumptions":{"status":"none_found","items":[]},"datasets":{"status":"available","items":[{"name":"SEC filings and financial documents vector database","provider":"SEC","kind":"text-news","access":"unknown","period":null,"universe":null,"quote":"For this experiment, a vector database containing processed SEC filings and financial documents was prepared; embeddings of document chunks were stored to allow efficient semantic retrieval when needed.","support":"full"},{"name":"Labeled dataset of financial news headlines with price movements","provider":null,"kind":"text-news","access":"unknown","period":null,"universe":null,"quote":"Using activations from the SAE of the Gemma-2B model and a labeled dataset of financial news headlines with corresponding price movements, we performed feature discovery to identify the most relevant features from the dataset.","support":"full"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"unclear"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"unclear","findings":[{"finding":"Mechanistic interpretability tools can isolate crucial risk or sentiment indicators and enable dynamic model adjustments, reducing the need for costly fine‑tuning.","kind":"other","quote":"Our experiments showed that these interpretability tools not only help isolate crucial risk or sentiment indicators but also allow dynamic model adjustments—cutting down on costly fine-tuning and enabling more precise responses in high-stakes scenarios.","support":"full"},{"finding":"Feature steering shifts model sentiment toward credit‑risk factors, improving classification balance without extensive prompt engineering.","kind":"performance","quote":"With feature steering applied at a high magnitude to the feature \"concepts related to credit ratings and financial recovery\", the sentiment shifts to 4, highlighting both the negative (lower credit score) and positive (secured financing & debt reduction) aspects, ultimately improving the overall outlook. This change occurs because steering shifts the model’s attention toward financially stabilizing factors, ensuring","support":"full"},{"finding":"Sparse autoencoder feature activations combined with a decision‑tree classifier achieve interpretable short‑term stock price movement predictions with measurable F1 performance.","kind":"predictive-accuracy","quote":"Once the key features were identified, we trained a decision tree classifier to find the optimal combination of features and evaluated its performance using the F1 score. The decision tree classifier, trained on this data, is visualized to show how different features contribute to predicting stock price movements (up or down).","support":"full"},{"finding":"Activation‑based feature magnitude can be used for bias detection in loan approval and for hallucination reduction via retrieval‑augmented generation.","kind":"other","quote":"6.2.2 Bias Detection and Reduction In this experiment, we address bias reduction in the loan approval process by analyzing feature activations in sparse autoencoders (SAE). 6.2.3 Hallucination Reduction In this experiment, we aimed to enhance the reliability of the language model’s output by mitigating potential hallucinations, specifically in the context of financial information.","support":"full"},{"finding":"Logit Lens visualizations reveal how probabilities shift across layers, clarifying which model parts influence specific financial predictions.","kind":"theoretical","quote":"5.1 Logit Lens In this experiment, Logit Lens is employed to track how probabilities shift across layers, allowing for a clearer understanding of which parts of the model influence specific financial predictions. First, we used the Logit Lens to generate a heatmap as shown in Figure 5 illustrating the model’s predicted token probabilities at each layer for a given financial prompt.","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[],"contradicts":[],"qualifies":[],"compared_with":21,"citations":{"cited_by_in_corpus":8,"by_independent_authors":8,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":2,"mentions":5,"unclassified":1},"independent_papers":[{"paper":{"paper_id":"2512.23515","title":"Alpha-R1: Alpha Screening with LLM Reasoning via Reinforcement Learning","paper_date":"2025-12-29","authors":["Zuoyou Jiang","Li Zhao","Rui Sun","Ruohan Sun","Zhongjian Li","Jing Li"],"empirical_rigor":8.5,"page_url":"https://thequant.space/flowcharts/alpha-r1-alpha-screening-with-llm-reasoning-via-reinforceme/"},"how":"discusses_findings","quote":"Neither sparsity-based methods [ref] , which exhibit instability during regime changes, nor unaligned general-purpose LLMs [CITED] currently meet this requirement."},{"paper":{"paper_id":"2602.07096","title":"RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?","paper_date":"2026-02-06","authors":["Yuyang Dai","Yan Lin","Zhuohan Xie","Yuxia Wang"],"empirical_rigor":6,"page_url":"https://thequant.space/flowcharts/realfin-how-well-do-llms-reason-about-finance-when-users-leave-things-unsaid/"},"how":"discusses_findings","quote":"Many models remain difficult to interpret and control in ways that satisfy regulatory expectations [CITED]."}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T19:50:10Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}