{"paper_id":"2310.04793","title":"FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets","authors":["Neng Wang","Hongyang Yang","Christina Dan Wang"],"paper_date":"2023-10-07","paper_type":"dataset-benchmark","paper_url":"https://arxiv.org/pdf/2310.04793v2","page_url":"https://thequant.space/flowcharts/fingpt-instruction-tuning-benchmark-for-open-source-large-l/","scores":{"empirical_rigor":7,"math_complexity":2.5,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes an Instruction Tuning paradigm tailored for open-source large language models in the financial sector.","quote":"In this paper, we propose a novel scheme that utilizes the Instruction Tuning paradigm to magnify the capabilities of LLMs within the financial sphere.","summary":"In the swiftly expanding domain of Natural Language Processing (NLP), the potential of GPT-based models for the financial sector is increasingly evident. However, the integration of these models with financial datasets presents challenges, notably in determining their adeptness and relevance. This…","strategy":null},"assumptions":{"status":"none_found","items":[]},"datasets":{"status":"available","items":[{"name":"FPB","provider":null,"kind":"text-news","access":"public","period":null,"universe":null,"quote":"For the Sentiment Analysis (SA) task, we leverage datasets FPB[17] and FiQA-SA[16]"},{"name":"FiQA-SA","provider":null,"kind":"text-news","access":"public","period":null,"universe":null,"quote":"For the Sentiment Analysis (SA) task, we leverage datasets FPB[17] and FiQA-SA[16]","known_id":"fiqa"},{"name":"TFNS","provider":null,"kind":"text-news","access":"public","period":null,"universe":null,"quote":"additional datasets, TFNS[15] and NWGI[36], were incorporated into the mix"},{"name":"NWGI","provider":null,"kind":"text-news","access":"public","period":null,"universe":null,"quote":"additional datasets, TFNS[15] and NWGI[36], were incorporated into the mix"},{"name":"Headline","provider":null,"kind":"text-news","access":"public","period":null,"universe":null,"quote":"for the Headline Classification (HC) task, the Headline dataset[25] is employed"},{"name":"FinRED","provider":null,"kind":"text-news","access":"public","period":null,"universe":null,"quote":"For this endeavor, we engaged with the FinRED dataset[24], providing a basis for the extraction and understanding of financial relations"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"none"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"Qwen consistently enhanced its performance across all sentiment analysis sub‑datasets in the multi‑task setting.","kind":"performance","quote":"An exception to this trend, Qwen consistently enhanced its performance across all SA sub-datasets."},{"finding":"Falcon excelled in task‑specific and multi‑task Instruction Tuning phases and showed strong generalization in zero‑shot tasks.","kind":"performance","quote":"Falcon excelled in the task-specific and multi-task Instruction Tuning phases but demonstrated significant generalization capabilities, understanding unseen instructions, and making accurate classifications during the zero-shot tasks."},{"finding":"MPT registered the most significant improvement in both NER and RE tasks after multi‑task tuning.","kind":"performance","quote":"MPT registered the most significant improvement in both NER and RE tasks"},{"finding":"BLOOM suffered the most substantial performance degradation in classification tasks after multi‑task tuning.","kind":"negative","quote":"BLOOM, which already had limited success during the task-specific tuning phase, suffered the most substantial performance degradation in classification tasks after multi-task tuning."},{"finding":"Overall, Llama2 and MPT displayed versatility by excelling in various tasks and benefiting from the multi‑task learning environment.","kind":"performance","quote":"Llama2 and MPT not only displayed versatility by excelling in various tasks but also benefited from the multi-task learning environment."}],"reported_performance":null,"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":16,"citations":{"cited_by_in_corpus":13,"by_independent_authors":7,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":6,"unclassified":7},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/AI4Finance-Foundation/FinGPT","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T18:20:57Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}