Paper: arXiv 2402.11728
Authors: Agam Shah, Arnav Hiray, Pratvi Shah, Arkaprabha Banerjee, Anushka Singh, Dheeraj Eidnani, Sahasra Chava, Bhaskar Chaudhury, Sudheer Chava
Abstract
In this paper, we investigate the influence of claims in analyst reports and earnings calls on financial market returns, considering them as significant quarterly events for publicly traded companies. To facilitate a comprehensive analysis, we construct a new financial dataset for the claim detection task in the financial domain. We benchmark various language models on this dataset and propose a novel weak-supervision model that incorporates the knowledge of subject matter experts (SMEs) in the aggregation function, outperforming existing approaches. We also demonstrate the practical utility of our proposed model by constructing a novel measure of optimism. Here, we observe the dependence of earnings surprise and return on our optimism measure. Our dataset, models, and code are publicly (under CC BY 4.0 license) available on GitHub.
Complexity vs Empirical Score
- Math Complexity: 2.5/10
- Empirical Rigor: 8.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper is primarily focused on NLP and dataset construction with benchmarking language models and public data release, showing high empirical rigor, while the mathematics involved (statistics for weak-supervision aggregation) is relatively low in complexity.
Research Flowchart
flowchart TD A["Research Goal: Determine influence of claims<br>on market returns & create claim detection model"] --> B["Create New Financial Dataset"] B --> C["Develop Novel Weak-Supervision Model<br>incorporating SME knowledge"] C --> D["Benchmark against other<br>Language Models"] D --> E["Construct Novel<br>Optimism Measure"] E --> F["Analyze Earnings Surprise<br>vs. Optimism"] F --> G["Key Findings: Claims impact markets,<br>Proposed model outperforms<br>Optimism correlates with earnings"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a novel weak‑supervision model with a custom aggregation function that uses subject‑matter expert knowledge, and a measure of optimism derived from it.
Source sentence
We benchmark various language models on this dataset and propose a novel weak-supervision model that incorporates the knowledge of subject matter experts (SMEs) in the aggregation function, outperforming existing approaches.
Assumes 5
- Trading decisions are based on the optimism measure without accounting for transaction costs. execution
Source sentence
We use Earnings Surprise, CAR[+2, +30], and CAR[+2,+60] to determine the success or failure of our hypothetical trades.
- The rule‑based strategy can predict stock direction with high accuracy. market
Source sentence
such a rule-based strategy achieves an approximate 81% accuracy in correctly predicting the direction of stock movement.
- Numeric sentences with a currency or percentage symbol are the relevant financial sentences. data
Source sentence
sentences with a numeric value AND either a currency symbol as a prefix or percentage symbol as a postfix‑are extracted, as they have numerical relevance
- Financial significance is determined by matching words to a dictionary of 8,200 terms. data
Source sentence
The financial dictionary used in this study, developed by Shah et al. (2022), contains over 8,200 financially significant terms.
Show 1 more
- The weak‑supervision model uses the same set of labeling functions across aggregation methods. model
Source sentence
For all three models, the same set of labeling functions is used and they only differ in the aggregation part.
Data
- Analyst Reports Dataset text-news public
Source sentence
Our dataset of raw analyst reports in the English Language from 1,530 major companies over the period of 2017-20 is significantly larger than NumClaim
- Earnings Call Transcripts text-news public
Source sentence
Collecting data till March of 2023 results in a total of 1,085 earnings call transcripts.
- Earnings Surprise Data fundamentals commercial
Source sentence
We collect the actual earnings per share (EPS) and forecasted median EPS from the I/B/E/S dataset.
- Financial Dictionary alternative public
Source sentence
The financial dictionary used in this study, developed by Shah et al. (2022), contains over 8,200 financially significant terms.
Costs
Not modeled
Finds 2
- The weak‑supervision model achieves 93% accuracy on claim detection. performance
Source sentence
Our work presents claim based labeled dataset in the English language alongside presenting a weak‑supervision model with an accuracy of 93%.
- A rule‑based trading strategy using the optimism measure predicts stock direction with about 81% accuracy up to 60 days. predictive-accuracy
Source sentence
such a rule-based strategy achieves an approximate 81% accuracy in correctly predicting the direction of stock movement.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 9 related papers). Coverage is partial, so this is neither replication nor refutation.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (63)
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023) — link
- Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis — → 2305.07972
- When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain (2022) — link
- Analysts’ forecast optimism: The effects of managers’ incentives on analysts’ forecasts (2022) — link
- WRENCH: A Comprehensive Benchmark for Weak Supervision (2021) — link
- FinQA: A Dataset of Numerical Reasoning over Financial Data (2021) — link
- Persuading Investors: A Video-Based Study (2021) — link
- Multimodal Machine Learning for Credit Modeling (2021) — link
and 55 more
- skweak: Weak Supervision Made Easy for NLP (2021) — link
- Green Algorithms: Quantifying the Carbon Footprint of Computation (2020) — link
- VolTAGE: Volatility Forecasting via Text-Audio Fusion with Graph Convolution Networks for Earnings Calls (2020) — link
- NumClaim: Investor’s Fine-grained Claim Detection (2020) — link
- MAEC: A Multimodal Aligned Earnings Conference Call Dataset for Financial Risk Prediction (2020) — link
- Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach (2020) — link
- How to Talk When a Machine is Listening: Corporate Disclosure in the Age of AI (2020) — link
- FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining (2020) — link
- Measuring Corporate Culture Using Machine Learning (2020) — link
- FinBERT: A Pretrained Language Model for Financial Communications (2020) — link
- Man Versus Machine: A Comparison of Robo-Analyst and Traditional Research Analyst Investment Recommendations (2020) — link
- PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019) — link
- FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019) — link
- Numeral Attachment with Auxiliary Tasks (2019) — link
- Snuba: Automating Weak Supervision to Label Training Data (2018) — link
- Weakly-Supervised Neural Text Classification (2018) — link
- WWW'18 Open Challenge: Financial Opinion Mining and Question Answering (2018) — link
- Snorkel: Rapid Training Data Creation with Weak Supervision (2017) — link
- A Multilayer Perceptron based Ensemble Technique for Fine-grained Financial Sentiment Analysis (2017) — link
- Attention is All you Need (2017) — link
- Deep learning for financial sentiment analysis on finance news providers (2016) — link
- GloVe: Global Vectors for Word Representation (2014) — link
- Good debt or bad debt: Detecting semantic orientations in economic texts (2013) — link
- A Survey of Crowdsourcing Systems (2011) — link
- A Survey on Transfer Learning (2010) — link
- Do Analysts Herd? An Analysis of Recommendations and Market Reactions (2010) — link
- Making Sense of Cents: An Examination of Firms That Marginally Miss or Beat Analyst Forecasts (2009) — link
- Distant supervision for relation extraction without labeled data (2009) — link
- Conflict of interest and the credibility of underwriter analyst recommendations (1999) — link
- Long Short-Term Memory (1997) — link
- Bidirectional recurrent neural networks (1997) — link
- ANALYSTS DECISIONS AS PRODUCTS OF A MULTITASK ENVIRONMENT (1993) — link
- Retail Investors and Analysts (2024) — link
- . Falcon-40B: an open large language model with state-of-the-art performance (2023)
- Do Managers Walk the Talk on Environmental and Social Issues? (2021) — link
- ReGAL: Rule-Generative Active Learning for Model-in-the-Loop Weak Supervision (2020)
- Transform-ers: State-of-the-art natural language processing (2020)
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
- Investment Banking Relationships and Analyst Affiliation Bias: The Impact of Global Settlement on Sanctioned and Non-Sanctioned Banks (2014)
- Semi-Supervised Learning (2009) — link
- Burr Settles (2009)
- Comparing the stock recommendation performance of investment banks and independent research firms (2006) — link
- Measuring firm-level inflation exposure: A deep learning approach
- Phrases often provided definitive information about a given sentence in a document and in most cases they had a fairly consistent linguistic composition
- 2023b. Finer: Financial named entity recognition dataset and weak-supervision model
- a bid to capture the effect of a few other verb forms indicative of a probabilistic event, we
- by us while choosing the rules to weak-supervision mode. All rules were post detailed analysis of sample
- no contamination issue in the weak-supervision model and it is generalizable, we get
- Chat-gpt in finance: Applications, challenges, and solutions
- Actune: Uncertainty-based active self-training for active fine-tuning of pretrained language models
- Closed ai models make bad baselines
- The alternate adoption of phrase matching was to identify in-claim sentences
- robustness of our model on the dataset annotated by annotators who don’t have knowledge of the rules used in the weak-supervision model
- Falcon-40B: an open large language model
1 of 63 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.