Paper: arXiv 2510.05710

Authors: Fabrizio Dimino, Abhinav Arun, Bhaskarjit Sarmah, Stefano Pasquali

Abstract

Large language models (LLMs) are increasingly being used to extract structured knowledge from unstructured financial text. Although prior studies have explored various extraction methods, there is no universal benchmark or unified evaluation framework for the construction of financial knowledge graphs (KG). We introduce FinReflectKG - EvalBench, a benchmark and evaluation framework for KG extraction from SEC 10-K filings. Building on the agentic and holistic evaluation principles of FinReflectKG - a financial KG linking audited triples to source chunks from S&P 100 filings and supporting single-pass, multi-pass, and reflection-agent-based extraction modes - EvalBench implements a deterministic commit-then-justify judging protocol with explicit bias controls, mitigating position effects, leniency, verbosity and world-knowledge reliance. Each candidate triple is evaluated with binary judgments of faithfulness, precision, and relevance, while comprehensiveness is assessed on a three-level ordinal scale (good, partial, bad) at the chunk level. Our findings suggest that, when equipped with explicit bias controls, LLM-as-Judge protocols provide a reliable and cost-efficient alternative to human annotation, while also enabling structured error analysis. Reflection-based extraction emerges as the superior approach, achieving best performance in comprehensiveness, precision, and relevance, while single-pass extraction maintains the highest faithfulness. By aggregating these complementary dimensions, FinReflectKG - EvalBench enables fine-grained benchmarking and bias-aware evaluation, advancing transparency and governance in financial AI applications.

Complexity vs Empirical Score

  • Math Complexity: 3.0/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper focuses on benchmarking methodology and evaluation protocols with minimal advanced mathematics, relying instead on descriptive statistics and scoring metrics, while demonstrating high empirical rigor through a reproducible benchmark on real SEC filings with bias controls and multiple extraction modes.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Develop unified KG<br>evaluation benchmark"] --> B["Data<br>SEC 10-K filings from<br>S&P 100 companies"]
  B --> C["Methodology<br>FinReflectKG EvalBench<br>Deterministic commit-then-justify protocol"]
  C --> D["Bias Controls<br>Position, Leniency,<br>Verbosity, World-Knowledge"]
  D --> E["Evaluation Dimensions<br>Faithfulness (Binary)<br>Precision (Binary)<br>Relevance (Binary)<br>Comprehensiveness (3-level)"]
  E --> F["Key Findings<br>1. Reflection extraction superior<br>2. LLM-as-Judge reliable with bias controls<br>3. Single-pass best for faithfulness"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes FinReflectKG‑EvalBench, a benchmark and bias‑aware evaluation framework for financial knowledge‑graph extraction from SEC 10‑K filings.

Source sentence
We introduce FinReflectKG‑EvalBench, a benchmark and bias‑aware evaluation framework for KG extraction from SEC 10‑K filings.

Assumes

No explicit assumptions found in the text.

Data

  • SEC 10‑K filings · U.S. SEC text-news public
    fiscal year 2024 · S&P 100 companies
    Source sentence
    FinReflectKG‑EvalBench is currently scoped to U.S. SEC Form 10‑K filings from S&P 100 companies for fiscal year 2024

Costs

Not applicable The paper reports no trading or execution result.

Finds 2

  • Reflection‑based extraction achieves the best performance in comprehensiveness, precision, and relevance, while single‑pass extraction maintains the highest faithfulness. performance
    Source sentence
    Reflection‑based extraction emerges as the superior approach, achieving best performance in comprehensiveness, precision, and relevance, while single‑pass extraction maintains the highest faithfulness.
  • Inter‑judge agreement is substantial across dimensions, with Krippendorff’s α of 0.88 for faithfulness, 0.84 for precision, 0.77 for relevance, and 0.71 for comprehensiveness. statistical
    Source sentence
    Agreement is lower for relevance (α = 0.77) and comprehensiveness (α = 0.71).

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 10 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (90)

  1. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning (2017) — link
  2. Democratizing Financial Knowledge Graph Construction by Mining Massive Brokerage Research Reports (2022)
  3. GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction (2024) — link
  4. Knowledge Graph Construction from 10-K Filings with Open-Domain Extraction (2024)
  5. Style Over Substance: Evaluation Biases for Large Language Models (2023)
  6. Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation (2025) — link
  7. Measuring Self-Preference in LLM Judgments (2025) — link
  8. Automated Annotation with Generative AI Requires Validation (2023) — link
and 82 more
  1. Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections (2025) — link
  2. Humans or LLMs as the Judge? A Study on Judgement Bias (2024)
  3. A Graph-Based Method for Unsupervised Knowledge Discovery from Financial Texts (2022) — link
  4. Data Set and Evaluation of Automated Construction of Financial Knowledge Graph (2021) — link
  5. The financial industry business ontology: Best practice for big data (2013) — link
  6. The Financial Industry Business Ontology (FIBO) — link
  7. SQLNet: Generating Structured Queries from Natural Language Without Reinforcement Learning (2017) — link
  8. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers (2019) — link
  9. Pointer Networks (2015) — link
  10. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) — link
  11. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (2018) — link
  12. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (2023)
  13. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022) — link
  14. Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022) — link
  15. Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrieval (2023) — link
  16. A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL (2024) — link
  17. MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL (2023) — link
  18. SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration (2024) — link
  19. Knowledge Graph Construction: Extraction, Learning, and Evaluation (2025) — link
  20. From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (2025) — link
  21. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation (2024) — link
  22. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models (2023) — link
  23. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020) — link
  24. Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction (2024) — link
  25. Reasoning or Overthinking: Evaluating Large Language Models on Financial Sentiment Analysis (2025) — link
  26. Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
  27. Do not think that much for 2+ 3=? on the overthinking of o1-like llms (2024) — link
  28. Stop overthinking: A survey on efficient reasoning for large language models (2025) — link
  29. Reflexion: an autonomous agent with dynamic memory and self-reflection (2024) — link
  30. Reflexion: Language Agents with Verbal Reinforcement Learning (2023) — link
  31. A neural multi-task learning framework to jointly model medical named entity recognition and normalization (2018)
  32. Joint learning of named entity recognition and disambiguation (2019)
  33. Relation classification via convolutional deep neural network (2014)
  34. Distant supervision for relation extraction via piecewise convolutional neural networks (2015)
  35. Evaluating ChatGPT’s Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness (2023) — link
  36. Iterative Zero-Shot Prompting for Knowledge Graph Construction with Large Language Models (2023) — link
  37. RAS: Retrieval-And-Structuring for Knowledge-Intensive Generation (2025) — link
  38. ChatIE: Language Model as an Interactive Information Extractor (2023) — link
  39. CESI: Canonicalizing Open Knowledge Bases using Embeddings and Side Information (2018)
  40. Financial news event extraction using semi-supervised learning (2017) — link
  41. A pipeline for financial knowledge graph construction from news (2020) — link
  42. Graphreader: Building graph-based agent to enhance long-context abilities of large language models (2024)
  43. Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering (2025) — link
  44. FinDKG: Dynamic Financial Knowledge Graph Construction from News with Large Language Models (2024)
  45. Leveraging Large Language Models to Improve Knowledge Graph Construction (2024)
  46. Structural Quality Metrics to Evaluate Knowledge Graph Quality (2022)
  47. Knowledge Graph Quality Control: A Survey (2021) — link
  48. Semantic-KG: Using Knowledge Graphs to Construct Benchmarks for Measuring Semantic Similarity (2025)
  49. Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models (2024) — link
  50. Docling: An Open-Source, Extensible and Efficient PDF Document Converter (2024) — link
  51. On measures of entropy and information (1961)
  52. A Mathematical Theory of Communication (1948)
  53. Revisiting Relation Extraction in the era of Large Language Models (2023) — link
  54. BizBench: A Quantitative Reasoning Benchmark for Business and Finance (2023) — link
  55. DocFinQA: A Long-Context Financial Reasoning Dataset (2024) — link
  56. FinKG: A Core Financial Knowledge Graph for Financial Analysis (2023) — link
  57. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods (2024) — link
  58. A survey on evaluation of large language models (2024)
  59. Bias and fairness in large language models: A survey (2024)
  60. Large language models are not robust multiple choice selectors (2023) — link
  61. DeepEval: An Open-Source Evaluation Framework for LLM Applications (2024)
  62. Ragas: Evaluation Framework for Retrieval-Augmented Generation (2023)
  63. RAGAS: Benchmarking Comprehensiveness in Retrieval-Augmented Generation (2023)
  64. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — link
  65. Llms-as-judges: a comprehensive survey on llm-based evaluation methods (2024) — link
  66. Tracing Positional Bias in Financial Decision-Making: Mechanistic Insights from Qwen2.5 — → 2508.18427
  67. AlpacaEval: An Automatic Evaluator of Instruction-Following Models (2023)
  68. Large Language Models are not Fair Evaluators (2024) — link
  69. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (2024) — link
  70. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge (2024) — link
  71. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — link
  72. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-Judge (2024) — link
  73. TruLens: Evaluation and Monitoring for LLM Applications (2023)
  74. LangSmith: Evaluation, Monitoring, and Tracing for LLM Applications (2023)
  75. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards (2024)
  76. LLM-as-a-judge: a complete guide to using LLMs for evaluations (2025)
  77. Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge (2024)
  78. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024)
  79. Can You Trick the Grader? Adversarial Persuasion of LLM Judges (2025)
  80. A High Precision Pipeline for Financial Knowledge Graph Construction (2020)
  81. FinReflectKG: Agentic construction and evaluation of financial knowledge graphs — → 2508.17906
  82. Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges (2025) — link

3 of 90 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.