Paper: arXiv 2310.10760
Authors: Bhaskarjit Sarmah, Tianjie Zhu, Dhagash Mehta, Stefano Pasquali
Abstract
For a financial analyst, the question and answer (Q&A) segment of the company financial report is a crucial piece of information for various analysis and investment decisions. However, extracting valuable insights from the Q&A section has posed considerable challenges as the conventional methods such as detailed reading and note-taking lack scalability and are susceptible to human errors, and Optical Character Recognition (OCR) and similar techniques encounter difficulties in accurately processing unstructured transcript text, often missing subtle linguistic nuances that drive investor decisions. Here, we demonstrate the utilization of Large Language Models (LLMs) to efficiently and rapidly extract information from earnings report transcripts while ensuring high accuracy transforming the extraction process as well as reducing hallucination by combining retrieval-augmented generation technique as well as metadata. We evaluate the outcomes of various LLMs with and without using our proposed approach based on various objective metrics for evaluating Q&A systems, and empirically demonstrate superiority of our method.
Complexity vs Empirical Score
- Math Complexity: 2.5/10
- Empirical Rigor: 7.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper focuses on applying established NLP techniques (LLMs, RAG) rather than deriving new complex mathematics, but it includes a detailed empirical study using a real-world dataset of 50 earnings transcripts and quantitative evaluation metrics, making it highly data/implementation-heavy.
Research Flowchart
flowchart TD
Start["Research Goal: Reduce hallucination in LLM-based financial Q&A extraction"] --> Method["Proposed Method: RAG + Metadata"]
subgraph Inputs ["Data & Inputs"]
Input1["Financial Report Transcripts"]
Input2["Q&A Question Pairs"]
end
Inputs --> Process1["Data Preprocessing & Chunking"]
Process1 --> Process2["Embedding Generation"]
Process2 --> Process3["Knowledge Base Construction"]
subgraph Computation ["LLM Processing"]
User["User Query"] --> RAG["Retrieval-Augmented Generation"]
Metadata["Financial Metadata Injection"] --> RAG
Process3 -.-> RAG
RAG --> LLM["Large Language Model Analysis"]
end
LLM --> Output["Extracted Insights & Responses"]
Output --> Evaluate["Evaluation Phase"]
Evaluate --> Metrics["Metrics: Accuracy, Relevance, Hallucination Rate"]
Metrics --> Compare["Comparison: LLM w/ vs w/o Method"]
Compare --> Result["Result: Superior Performance & Reduced Hallucination"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a retrieval‑augmented generation approach combined with metadata to reduce hallucinations in LLM‑based Q&A extraction from earnings reports.
Source sentence
We take a comprehensive approach to address this multifaceted challenge, by not only integrating retrieval‑augmented LLMs but also leveraging metadata to effectively mitigate the occurrence of such hallucinatory responses.
Assumes 5
- The earnings call transcripts from Nifty 50 companies are representative of financial Q&A content. data
Source sentence
We used transcripts from earnings calls of Nifty 50 constituents for our analysis.
- The text‑bison model’s context window (8192 tokens) limits direct processing of full transcripts, requiring chunking. model
Source sentence
text‑bison model has context window of 8192 tokens which is equivalent to 7000 words approximately and output token of 1024 tokens
- Chunking large documents into smaller segments preserves the necessary context for accurate Q&A. model
Source sentence
Chunking entails breaking down the document into smaller, more manageable segments that comfortably fit within the context window of the expansive language model
- Metadata associated with each transcript segment can be used to filter relevant chunks for a given company query. model
Source sentence
When responding to user queries about a particular company, we leveraged the metadata information to selectively filter and retrieve only those document segments pertinent to the queried company.
Show 1 more
- Retrieval‑augmented generation reduces hallucination compared to vanilla LLM generation. model
Source sentence
retrieval‑augmented LLMs aim to bolster the system’s accuracy by harnessing external repositories of information
Data
- Nifty 50 earnings call transcripts Q1 FY2024 text-news public
Source sentence
focus on data spanning the quarter ending in June, 2023 i.e. the earnings reports for Q1 of the financial year 20241... Our dataset encompasses 50 transcripts for this quarter, spanning over 50 companies within Nifty 50 universe
Costs
Not applicable
Finds 2
- The retrieval‑augmented generation with metadata significantly improves answer accuracy and relevance compared to baseline LLMs, as measured by multiple evaluation metrics on 400 manually curated Q&A pairs. performance
Source sentence
This approach enhanced the accuracy and relevance of our generated answers as measured by various evaluation metrics and with respect to the manually curated, ground‑truth, answers to 400 questions.
- The proposed method outperforms other LLMs on semantic similarity metrics, with PaLM2 achieving the best BERT Score and Pythia the best BART Score. statistical
Source sentence
the PaLM2 model outperforms all others for BERT Score while Pythia performs best in BART Score
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 7 related papers). Coverage is partial, so this is neither replication nor refutation.
5 papers in the archive : Quantifying Qualitative Insights: Leveraging LLMs to Market …
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.