{"paper_id":"2310.10760","title":"Towards reducing hallucination in extracting information from financial reports using Large Language Models","authors":["Bhaskarjit Sarmah","Tianjie Zhu","Dhagash Mehta","Stefano Pasquali"],"paper_date":"2023-10-16","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2310.10760v1","page_url":"https://thequant.space/flowcharts/towards-reducing-hallucination-in-extracting-information-fro/","scores":{"empirical_rigor":7,"math_complexity":2.5,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes a retrieval‑augmented generation approach combined with metadata to reduce hallucinations in LLM‑based Q&A extraction from earnings reports.","quote":"We take a comprehensive approach to address this multifaceted challenge, by not only integrating retrieval‑augmented LLMs but also leveraging metadata to effectively mitigate the occurrence of such hallucinatory responses.","summary":"For a financial analyst, the question and answer (Q&A) segment of the company financial report is a crucial piece of information for various analysis and investment decisions. However, extracting valuable insights from the Q&A section has posed considerable challenges as the conventional methods…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The earnings call transcripts from Nifty 50 companies are representative of financial Q&A content.","type":"data","stated":"explicit","quote":"We used transcripts from earnings calls of Nifty 50 constituents for our analysis."},{"assumption":"The text‑bison model’s context window (8192 tokens) limits direct processing of full transcripts, requiring chunking.","type":"model","stated":"explicit","quote":"text‑bison model has context window of 8192 tokens which is equivalent to 7000 words approximately and output token of 1024 tokens"},{"assumption":"Chunking large documents into smaller segments preserves the necessary context for accurate Q&A.","type":"model","stated":"implied","quote":"Chunking entails breaking down the document into smaller, more manageable segments that comfortably fit within the context window of the expansive language model"},{"assumption":"Metadata associated with each transcript segment can be used to filter relevant chunks for a given company query.","type":"model","stated":"explicit","quote":"When responding to user queries about a particular company, we leveraged the metadata information to selectively filter and retrieve only those document segments pertinent to the queried company."},{"assumption":"Retrieval‑augmented generation reduces hallucination compared to vanilla LLM generation.","type":"model","stated":"explicit","quote":"retrieval‑augmented LLMs aim to bolster the system’s accuracy by harnessing external repositories of information"}]},"datasets":{"status":"available","items":[{"name":"Nifty 50 earnings call transcripts Q1 FY2024","provider":"Company websites (Nifty 50 constituents)","kind":"text-news","access":"public","period":"Q1 FY2024 (April‑June 2023)","universe":"Nifty 50 constituents","quote":"focus on data spanning the quarter ending in June, 2023 i.e. the earnings reports for Q1 of the financial year 20241... Our dataset encompasses 50 transcripts for this quarter, spanning over 50 companies within Nifty 50 universe"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"unclear"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"The retrieval‑augmented generation with metadata significantly improves answer accuracy and relevance compared to baseline LLMs, as measured by multiple evaluation metrics on 400 manually curated Q&A pairs.","kind":"performance","quote":"This approach enhanced the accuracy and relevance of our generated answers as measured by various evaluation metrics and with respect to the manually curated, ground‑truth, answers to 400 questions."},{"finding":"The proposed method outperforms other LLMs on semantic similarity metrics, with PaLM2 achieving the best BERT Score and Pythia the best BART Score.","kind":"statistical","quote":"the PaLM2 model outperforms all others for BERT Score while Pythia performs best in BART Score"}],"reported_performance":null,"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[],"contradicts":[],"qualifies":[],"compared_with":7,"citations":{"cited_by_in_corpus":5,"by_independent_authors":5,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":1,"mentions":2,"unclassified":2},"independent_papers":[{"paper":{"paper_id":"2411.08404","title":"Quantifying Qualitative Insights: Leveraging LLMs to Market Predict","paper_date":"2024-11-13","authors":["Hoyoung Lee","Youngsoo Choi","Yuhee Kwon"],"empirical_rigor":6.5,"page_url":"https://thequant.space/flowcharts/quantifying-qualitative-insights-leveraging-llms-to-market/"},"how":"discusses_findings","quote":"LLMs improve the process of extracting information from the Q&A portions of financial reports, reducing hallucinations and increasing both the accuracy and scalability of insights [CITED]."}],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T18:20:36Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}