Paper: arXiv 2407.18327
Authors: Adria Pop, Jan Spörer
Abstract
This research dissects financial equity research reports (ERRs) by mapping their content into categories. There is insufficient empirical analysis of the questions answered in ERRs. In particular, it is not understood how frequently certain information appears, what information is considered essential, and what information requires human judgment to distill into an ERR. The study analyzes 72 ERRs sentence-by-sentence, classifying their 4940 sentences into 169 unique question archetypes. We did not predefine the questions but derived them solely from the statements in the ERRs. This approach provides an unbiased view of the content of the observed ERRs. Subsequently, we used public corporate reports to classify the questions’ potential for automation. Answers were labeled “text-extractable” if the answers to the question were accessible in corporate reports. 78.7% of the questions in ERRs can be automated. Those automatable question consist of 48.2% text-extractable (suited to processing by large language models, LLMs) and 30.5% database-extractable questions. Only 21.3% of questions require human judgment to answer. We empirically validate using Llama-3-70B and GPT-4-turbo-2024-04-09 that recent advances in language generation and information extraction enable the automation of approximately 80% of the statements in ERRs. Surprisingly, the models complement each other’s strengths and weaknesses well. The research confirms that the current writing process of ERRs can likely benefit from additional automation, improving quality and efficiency. The research thus allows us to quantify the potential impacts of introducing large language models in the ERR writing process. The full question list, including the archetypes and their frequency, will be made available online after peer review.
Complexity vs Empirical Score
- Math Complexity: 2.0/10
- Empirical Rigor: 7.5/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper relies on statistical counting and text classification rather than advanced mathematical modeling, scoring low on math complexity. It demonstrates high empirical rigor by manually analyzing 72 reports, deriving categories from data, and validating automation potential using specific LLMs (Llama-3-70B and GPT-4) on real financial data.
Research Flowchart
flowchart TD
A["<b>Research Goal</b><br>Quantify automation potential of Equity Research Reports using LLMs"] --> B
subgraph B ["Data & Methodology"]
B1["Dataset<br>72 Equity Research Reports"] --> B2["Sentence Classification<br>4,940 sentences into 169 unique question archetypes"]
B2 --> B3["Automation Assessment<br>Labeling questions as text-extractable, database-extractable, or requiring human judgment"]
end
B3 --> C["<b>Computational Validation</b><br>Testing with Llama-3-70B & GPT-4-turbo"]
C --> D["<b>Key Findings</b>"]
subgraph D ["Outcomes"]
D1["78.7% of questions are automatable"]
D2["48.2% text-extractable (LLM-suitable)"]
D3["30.5% database-extractable"]
D4["21.3% require human judgment"]
end
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes to automate equity research report statements using large language models, specifically validating Llama‑3‑70B and GPT‑4‑turbo for extracting answers to questions.
Source sentence
We empirically validate using Llama-3-70B and GPT-4-turbo-2024-04-09 that recent advances in language generation and information extraction enable the automation of approximately 80% of the statements in ERRs.
Assumes 2
- The language models are run with temperature set to zero to obtain deterministic best‑guess outputs. model
Source sentence
We set the temperatures of both models to zero (giving the models the chance to always use their true best guesses) and do not limit the number of output tokens.
- A question is considered "extractable" if its answer can be found in a single text source; otherwise it is "database‑extractable" or "non‑extractable". data
Source sentence
If a direct answer was matched in a single text source, the extractability was marked as “extractable.” If the answer was found in a financial markets database (such as Bloomberg), the question was marked as “database-extractable.” If not found in single text passage or database, the extractability was marked as “non-e
Data
- Equity Research Reports (ERRs) other commercial
Source sentence
We downloaded 72 ERRs dated from 2018 to 2023 from Bloomberg and Refinitiv Eikon. Each report had an average of seven pages, and the median is 6.8 pages per report.
Refinitiv / LSEG (Datastream, Eikon, Tick History) Bloomberg
Costs
Not applicable
Finds 3
- 78.7% of the questions in ERRs can be automated, with 48.2% text‑extractable and 30.5% database‑extractable; only 21.3% require human judgment. performance
Source sentence
78.7% of the questions in ERRs can be automated. Those automatable question consist of 48.2% text-extractable (suited to processing by large language models, LLMs) and 30.5% database-extractable questions. Only 21.3% of questions require human judgment to answer.
- Llama‑3‑70B extracted correct information from annual reports for 27% of 200 sampled questions, while GPT‑4‑turbo did so for 26%. performance
Source sentence
Our results show that Llama-3-70B is able to extract information from annual reports for 27% of the 200 questions. GPT-4 is able to extract the correct answer in 26% of the questions (figure 5).
- Model errors rarely overlap, suggesting ensemble use can improve automation coverage. other
Source sentence
Another finding is that model errors often do not overlap (figure 6). Language models for information extraction show promising performance for extracting financial data.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 9 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (65)
- Language Models are Few-Shot Learners (2020)
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019)
- Introduction to Receiver Operator Curves (2006)
- VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text (2014)
- Large Language Models are Zero-Shot Reasoners (2022)
- Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023)
- WWW'18 Open Challenge: Financial Opinion Mining and Question Answering (2018)
- Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts (2014)
and 57 more
- FinBERT: A Pre-Trained Financial Language Representation Model for Financial Text Mining (2021)
- Central Bank Communication and Policy Effectiveness (2005)
- The Long-Run Information Effect of Central Bank Communication (2019)
- FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019)
- Shallow Discourse Parsing for Open Information Extraction and Text Simplification (2022)
- Studying the Effect of Syntactic Simplification on Text Summarization (2023)
- Graphene: a Context-Preserving Open Information Extraction System (2018)
- Deep Learning-Based Question Answering: A Survey (2023)
- Atlas: Few-Shot Learning with Retrieval Augmented Language Models (2022)
- REALM: Retrieval -Augmented Language Model Pre-Training (2020)
- Dense Passage Retrieval for Open-Domain Question Answering (2020)
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
- LLaMA : Open and Efficient Foundation Language Models (2023)
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2022)
- A Survey of Advanced Methods for Efficient Text Summarization (2023)
- Extending Context Window of Large Language Models via Positional Interpolation (2023)
- BloombergGPT: A Large Language Model for Finance (2023)
- Choosing Factors (2018)
- A Five-Factor Asset Pricing Model (2015)
- Size and Book-to-Market Factors in Earnings and Returns (1995)
- The Behavior of Stock-Market Prices (1965)
- The Cross-Section of Expected Stock Returns (1992)
- On Persistence in Mutual Fund Performance (1997)
- Value and Momentum Everywhere (2013)
- Anonymous Equity Research (2021)
- Valuation Model Use and the Price Target Performance of Sell-Side Equity Analysts (2013)
- Information Content of Equity Analyst Reports (2005)
- Equity Valuation Models and Target Price Accuracy in Europe: Evidence From Equity Reports (2013)
- When Do Sell-Side Analyst Reports Really Matter? Shareholder Protection, Institutional Investors and the Informativeness of Equity Research (2015)
- Target Price Accuracy in Equity Research (2010)
- Reading Between the Lines: An Empirical Examination of Qualitative Attributes of Financial Analysts’ Reports (2012)
- A Corpus Study of Bank Financial Analyst Reports: Semantic Fields and Metaphors (2017)
- Do Security Analysts Exhibit Persistent Differences in Stock Picking Ability? (2004)
- Can Investors Profit From the Prophets? Security Analyst Recommendations and Stock Returns (2001)
- Stock Prices and Financial Analysts’ Recommendations (1983)
- Discrete Expectational Data and Portfolio Performance (1986)
- Stock Price Reactions to the Wall Street Journal ’s Securities Recommendations (1990)
- Stock Prices and the Dissemination of Analysts’ Recommendation (1991)
- The Anatomy of the Performance of Buy and Sell Recommendations (1995)
- Conflict of Interest and the Credibility of Underwriter Analyst Recommendations (1999)
- Bloom: A 176b-Parameter Open-Access Multilingual Language Model (2022)
- Scaling Language Models: Methods, Analysis & Insights from Training Gopher (2022)
- OPT : Open Pre-Trained Transformer Language Models (2022)
- Trends in AI Inference Energy Consumption: Beyond the Performance-vs-Parameter Laws of Deep Learning (2023)
- F in QA : A Dataset of Numerical Reasoning over Financial Data (2021)
- Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models (2023)
- Do Brokerage Analysts’ Recommendations Have Investment Value? (1996)
- Do Sell-Side Analysts Exhibit Differential Target Price Forecasting Ability? (2013)
- Improving Language Models by Retrieving From Trillions of Tokens (2021)
- Survey of Hallucination in Natural Language Generation (2023)
- TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance (2021)
- Human Versus Machine: A Comparison of Robo-Analyst and Traditional Research Analyst Investment Recommendations (2022)
- Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts (2014)
- Semantically Enhanced Passage Retrieval for Business Analysis Activity (2008)
- Financial Statement Analysis with Large Language Models — → 2407.17866
- The Llama 3 Herd of Models (2024)
- GPT-4 Technical Report (2023)
1 of 65 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.