{"paper_id":"2407.18327","title":"The Structure of Financial Equity Research Reports -- Identification of the Most Frequently Asked Questions in Financial Analyst Reports to Automate Equity Research Using Llama 3 and GPT-4","authors":["Adria Pop","Jan Spörer"],"paper_date":"2024-07-04","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2407.18327v2","page_url":"https://thequant.space/flowcharts/the-structure-of-financial-equity-research-reports----identi/","scores":{"empirical_rigor":7.5,"math_complexity":2,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes to automate equity research report statements using large language models, specifically validating Llama‑3‑70B and GPT‑4‑turbo for extracting answers to questions.","quote":"We empirically validate using Llama-3-70B and GPT-4-turbo-2024-04-09 that recent advances in language generation and information extraction enable the automation of approximately 80% of the statements in ERRs.","summary":"This research dissects financial equity research reports (ERRs) by mapping their content into categories. There is insufficient empirical analysis of the questions answered in ERRs. In particular, it is not understood how frequently certain information appears, what information is considered…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The language models are run with temperature set to zero to obtain deterministic best‑guess outputs.","type":"model","stated":"explicit","quote":"We set the temperatures of both models to zero (giving the models the chance to always use their true best guesses) and do not limit the number of output tokens."},{"assumption":"A question is considered \"extractable\" if its answer can be found in a single text source; otherwise it is \"database‑extractable\" or \"non‑extractable\".","type":"data","stated":"explicit","quote":"If a direct answer was matched in a single text source, the extractability was marked as “extractable.” If the answer was found in a financial markets database (such as Bloomberg), the question was marked as “database-extractable.” If not found in single text passage or database, the extractability was marked as “non-e"}]},"datasets":{"status":"available","items":[{"name":"Equity Research Reports (ERRs)","provider":"Bloomberg and Refinitiv Eikon","kind":"other","access":"commercial","period":"2018-2023","universe":"72 reports from 23 research providers","quote":"We downloaded 72 ERRs dated from 2018 to 2023 from Bloomberg and Refinitiv Eikon. Each report had an average of seven pages, and the median is 6.8 pages per report.","known_id":"bloomberg"}],"known":[{"id":"refinitiv","name":"Refinitiv / LSEG (Datastream, Eikon, Tick History)","provider":"LSEG","kind":"market-prices","access":"commercial","quote":"DATA We downloaded 72 ERRs dated from 2018 to 2023 from Bloomberg and Refinitiv Eikon.","mentions":3,"used_in":1},{"id":"bloomberg","name":"Bloomberg","provider":"Bloomberg L.P.","kind":"market-prices","access":"commercial","quote":"DATA We downloaded 72 ERRs dated from 2018 to 2023 from Bloomberg and Refinitiv Eikon.","mentions":2,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"none"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"unclear","findings":[{"finding":"78.7% of the questions in ERRs can be automated, with 48.2% text‑extractable and 30.5% database‑extractable; only 21.3% require human judgment.","kind":"performance","quote":"78.7% of the questions in ERRs can be automated. Those automatable question consist of 48.2% text-extractable (suited to processing by large language models, LLMs) and 30.5% database-extractable questions. Only 21.3% of questions require human judgment to answer."},{"finding":"Llama‑3‑70B extracted correct information from annual reports for 27% of 200 sampled questions, while GPT‑4‑turbo did so for 26%.","kind":"performance","quote":"Our results show that Llama-3-70B is able to extract information from annual reports for 27% of the 200 questions. GPT-4 is able to extract the correct answer in 26% of the questions (figure 5)."},{"finding":"Model errors rarely overlap, suggesting ensemble use can improve automation coverage.","kind":"other","quote":"Another finding is that model errors often do not overlap (figure 6). Language models for information extraction show promising performance for extracting financial data."}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":9,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T18:06:46Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}