Paper: arXiv 2510.05710
Authors: Fabrizio Dimino, Abhinav Arun, Bhaskarjit Sarmah, Stefano Pasquali
Abstract
Large language models (LLMs) are increasingly being used to extract structured knowledge from unstructured financial text. Although prior studies have explored various extraction methods, there is no universal benchmark or unified evaluation framework for the construction of financial knowledge graphs (KG). We introduce FinReflectKG - EvalBench, a benchmark and evaluation framework for KG extraction from SEC 10-K filings. Building on the agentic and holistic evaluation principles of FinReflectKG - a financial KG linking audited triples to source chunks from S&P 100 filings and supporting single-pass, multi-pass, and reflection-agent-based extraction modes - EvalBench implements a deterministic commit-then-justify judging protocol with explicit bias controls, mitigating position effects, leniency, verbosity and world-knowledge reliance. Each candidate triple is evaluated with binary judgments of faithfulness, precision, and relevance, while comprehensiveness is assessed on a three-level ordinal scale (good, partial, bad) at the chunk level. Our findings suggest that, when equipped with explicit bias controls, LLM-as-Judge protocols provide a reliable and cost-efficient alternative to human annotation, while also enabling structured error analysis. Reflection-based extraction emerges as the superior approach, achieving best performance in comprehensiveness, precision, and relevance, while single-pass extraction maintains the highest faithfulness. By aggregating these complementary dimensions, FinReflectKG - EvalBench enables fine-grained benchmarking and bias-aware evaluation, advancing transparency and governance in financial AI applications.
Complexity vs Empirical Score
- Math Complexity: 3.0/10
- Empirical Rigor: 7.5/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper focuses on benchmarking methodology and evaluation protocols with minimal advanced mathematics, relying instead on descriptive statistics and scoring metrics, while demonstrating high empirical rigor through a reproducible benchmark on real SEC filings with bias controls and multiple extraction modes.
Research Flowchart
flowchart TD A["Research Goal<br>Develop unified KG<br>evaluation benchmark"] --> B["Data<br>SEC 10-K filings from<br>S&P 100 companies"] B --> C["Methodology<br>FinReflectKG EvalBench<br>Deterministic commit-then-justify protocol"] C --> D["Bias Controls<br>Position, Leniency,<br>Verbosity, World-Knowledge"] D --> E["Evaluation Dimensions<br>Faithfulness (Binary)<br>Precision (Binary)<br>Relevance (Binary)<br>Comprehensiveness (3-level)"] E --> F["Key Findings<br>1. Reflection extraction superior<br>2. LLM-as-Judge reliable with bias controls<br>3. Single-pass best for faithfulness"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes FinReflectKG‑EvalBench, a benchmark and bias‑aware evaluation framework for financial knowledge‑graph extraction from SEC 10‑K filings.
Source sentence
We introduce FinReflectKG‑EvalBench, a benchmark and bias‑aware evaluation framework for KG extraction from SEC 10‑K filings.
Assumes
No explicit assumptions found in the text.
Data
- SEC 10‑K filings text-news public
Source sentence
FinReflectKG‑EvalBench is currently scoped to U.S. SEC Form 10‑K filings from S&P 100 companies for fiscal year 2024
Costs
Not applicable
Finds 2
- Reflection‑based extraction achieves the best performance in comprehensiveness, precision, and relevance, while single‑pass extraction maintains the highest faithfulness. performance
Source sentence
Reflection‑based extraction emerges as the superior approach, achieving best performance in comprehensiveness, precision, and relevance, while single‑pass extraction maintains the highest faithfulness.
- Inter‑judge agreement is substantial across dimensions, with Krippendorff’s α of 0.88 for faithfulness, 0.84 for precision, 0.77 for relevance, and 0.71 for comprehensiveness. statistical
Source sentence
Agreement is lower for relevance (α = 0.77) and comprehensiveness (α = 0.71).
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 10 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (90)
- Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning (2017) — link
- Democratizing Financial Knowledge Graph Construction by Mining Massive Brokerage Research Reports (2022)
- GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction (2024) — link
- Knowledge Graph Construction from 10-K Filings with Open-Domain Extraction (2024)
- Style Over Substance: Evaluation Biases for Large Language Models (2023)
- Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation (2025) — link
- Measuring Self-Preference in LLM Judgments (2025) — link
- Automated Annotation with Generative AI Requires Validation (2023) — link
and 82 more
- Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections (2025) — link
- Humans or LLMs as the Judge? A Study on Judgement Bias (2024)
- A Graph-Based Method for Unsupervised Knowledge Discovery from Financial Texts (2022) — link
- Data Set and Evaluation of Automated Construction of Financial Knowledge Graph (2021) — link
- The financial industry business ontology: Best practice for big data (2013) — link
- The Financial Industry Business Ontology (FIBO) — link
- SQLNet: Generating Structured Queries from Natural Language Without Reinforcement Learning (2017) — link
- RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers (2019) — link
- Pointer Networks (2015) — link
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) — link
- Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (2018) — link
- Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (2023)
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022) — link
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022) — link
- Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrieval (2023) — link
- A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL (2024) — link
- MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL (2023) — link
- SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration (2024) — link
- Knowledge Graph Construction: Extraction, Learning, and Evaluation (2025) — link
- From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (2025) — link
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation (2024) — link
- Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models (2023) — link
- Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (2020) — link
- Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction (2024) — link
- Reasoning or Overthinking: Evaluating Large Language Models on Financial Sentiment Analysis (2025) — link
- Can chatgpt forecast stock price movements? return predictability and large language models — → 2304.07619
- Do not think that much for 2+ 3=? on the overthinking of o1-like llms (2024) — link
- Stop overthinking: A survey on efficient reasoning for large language models (2025) — link
- Reflexion: an autonomous agent with dynamic memory and self-reflection (2024) — link
- Reflexion: Language Agents with Verbal Reinforcement Learning (2023) — link
- A neural multi-task learning framework to jointly model medical named entity recognition and normalization (2018)
- Joint learning of named entity recognition and disambiguation (2019)
- Relation classification via convolutional deep neural network (2014)
- Distant supervision for relation extraction via piecewise convolutional neural networks (2015)
- Evaluating ChatGPT’s Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness (2023) — link
- Iterative Zero-Shot Prompting for Knowledge Graph Construction with Large Language Models (2023) — link
- RAS: Retrieval-And-Structuring for Knowledge-Intensive Generation (2025) — link
- ChatIE: Language Model as an Interactive Information Extractor (2023) — link
- CESI: Canonicalizing Open Knowledge Bases using Embeddings and Side Information (2018)
- Financial news event extraction using semi-supervised learning (2017) — link
- A pipeline for financial knowledge graph construction from news (2020) — link
- Graphreader: Building graph-based agent to enhance long-context abilities of large language models (2024)
- Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering (2025) — link
- FinDKG: Dynamic Financial Knowledge Graph Construction from News with Large Language Models (2024)
- Leveraging Large Language Models to Improve Knowledge Graph Construction (2024)
- Structural Quality Metrics to Evaluate Knowledge Graph Quality (2022)
- Knowledge Graph Quality Control: A Survey (2021) — link
- Semantic-KG: Using Knowledge Graphs to Construct Benchmarks for Measuring Semantic Similarity (2025)
- Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models (2024) — link
- Docling: An Open-Source, Extensible and Efficient PDF Document Converter (2024) — link
- On measures of entropy and information (1961)
- A Mathematical Theory of Communication (1948)
- Revisiting Relation Extraction in the era of Large Language Models (2023) — link
- BizBench: A Quantitative Reasoning Benchmark for Business and Finance (2023) — link
- DocFinQA: A Long-Context Financial Reasoning Dataset (2024) — link
- FinKG: A Core Financial Knowledge Graph for Financial Analysis (2023) — link
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods (2024) — link
- A survey on evaluation of large language models (2024)
- Bias and fairness in large language models: A survey (2024)
- Large language models are not robust multiple choice selectors (2023) — link
- DeepEval: An Open-Source Evaluation Framework for LLM Applications (2024)
- Ragas: Evaluation Framework for Retrieval-Augmented Generation (2023)
- RAGAS: Benchmarking Comprehensiveness in Retrieval-Augmented Generation (2023)
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — link
- Llms-as-judges: a comprehensive survey on llm-based evaluation methods (2024) — link
- Tracing Positional Bias in Financial Decision-Making: Mechanistic Insights from Qwen2.5 — → 2508.18427
- AlpacaEval: An Automatic Evaluator of Instruction-Following Models (2023)
- Large Language Models are not Fair Evaluators (2024) — link
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (2024) — link
- Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge (2024) — link
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — link
- From Generation to Judgment: Opportunities and Challenges of LLM-as-a-Judge (2024) — link
- TruLens: Evaluation and Monitoring for LLM Applications (2023)
- LangSmith: Evaluation, Monitoring, and Tracing for LLM Applications (2023)
- When benchmarks are targets: Revealing the sensitivity of large language model leaderboards (2024)
- LLM-as-a-judge: a complete guide to using LLMs for evaluations (2025)
- Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge (2024)
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024)
- Can You Trick the Grader? Adversarial Persuasion of LLM Judges (2025)
- A High Precision Pipeline for Financial Knowledge Graph Construction (2020)
- FinReflectKG: Agentic construction and evaluation of financial knowledge graphs — → 2508.17906
- Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges (2025) — link
3 of 90 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.