{"paper_id":"2510.05710","title":"FinReflectKG - EvalBench: Benchmarking Financial KG with Multi-Dimensional Evaluation","authors":["Fabrizio Dimino","Abhinav Arun","Bhaskarjit Sarmah","Stefano Pasquali"],"paper_date":"2025-10-07","paper_type":"dataset-benchmark","paper_url":"https://arxiv.org/pdf/2510.05710v1","page_url":"https://thequant.space/flowcharts/finreflectkg---evalbench-benchmarking-financial-kg-with-mul/","scores":{"empirical_rigor":7.5,"math_complexity":3,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes FinReflectKG‑EvalBench, a benchmark and bias‑aware evaluation framework for financial knowledge‑graph extraction from SEC 10‑K filings.","quote":"We introduce FinReflectKG‑EvalBench, a benchmark and bias‑aware evaluation framework for KG extraction from SEC 10‑K filings.","summary":"Large language models (LLMs) are increasingly being used to extract structured knowledge from unstructured financial text. Although prior studies have explored various extraction methods, there is no universal benchmark or unified evaluation framework for the construction of financial knowledge…","strategy":null},"assumptions":{"status":"none_found","items":[]},"datasets":{"status":"available","items":[{"name":"SEC 10‑K filings","provider":"U.S. SEC","kind":"text-news","access":"public","period":"fiscal year 2024","universe":"S&P 100 companies","quote":"FinReflectKG‑EvalBench is currently scoped to U.S. SEC Form 10‑K filings from S&P 100 companies for fiscal year 2024"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"none"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"unclear","findings":[{"finding":"Reflection‑based extraction achieves the best performance in comprehensiveness, precision, and relevance, while single‑pass extraction maintains the highest faithfulness.","kind":"performance","quote":"Reflection‑based extraction emerges as the superior approach, achieving best performance in comprehensiveness, precision, and relevance, while single‑pass extraction maintains the highest faithfulness."},{"finding":"Inter‑judge agreement is substantial across dimensions, with Krippendorff’s α of 0.88 for faithfulness, 0.84 for precision, 0.77 for relevance, and 0.71 for comprehensiveness.","kind":"statistical","quote":"Agreement is lower for relevance (α = 0.77) and comprehensiveness (α = 0.71)."}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":10,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:25:06Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}