Paper: arXiv 2609.35744

Authors: Hoyoung Lee, Suyeol Yun, Jack Haverty, Yunju Cho, Meesong Kim, Daekyung Park, Sumin Kim, Jihoon Kwon, Jasmine Jia Geng, Andrew Chin, Yin Luo, Edward Tong, Yu Yu, Zach Golkhou, Minkyu Kim, Igor Halperin, Young Cha, Alejandro Lopez-Lira, Chanyeol Choi, Yongjae Lee

Abstract

Evaluating finance research agents requires rubrics that reflect expert standards and fix the values correct as of an information cutoff. Expert-reviewed finance benchmarks rely on fixed, per-item rubrics, which are costly to extend and cannot encode each institution’s own standard. In FinAutoRubric, experts specify reusable evaluation guidance, while agents and code carry out query-specific rubric generation, review, and validation. This expert guidance governs every agent, as prompts and as rules that code enforces, and a Task Bank of reusable criteria carries it across tasks. In long-horizon loops that follow the expert guidance, a writer agent researches every expected value and a reviewer agent verifies it, and failures escalate to a human. On three expert-authored finance benchmarks, its rubrics track expert scoring as closely as the strongest evaluated generator while stating the expert rubric’s expected value for more criteria, their scores agree with human grading, and in-house analysts prefer them in a blind review. The released 100-query FinAutoRubric Benchmark, built from in-house analysts’ key questions across 78 tasks and eight asset classes, shows that rubrics from an earlier model generation still leave headroom for a later one.

Complexity vs Empirical Score

  • Math Complexity: 0.0/10
  • Empirical Rigor: 0.0/10
  • Quadrant: Philosophers — conceptual discussion, lower math and empirical scores

Why this score: Scoring failed: Could not find valid JSON in LLM response: line 1 column 1 (char 0)

Research Flowchart

  flowchart TD
    A["Research Goal: Evaluate Financial Research Agents with Expert-Standard Rubrics"] --> B{"Key Methodology Steps"};
    B -- Expert Input --> C["Reusable Evaluation Guidance Specification"];
    B -- Agent-Driven --> D["Query-Specific Rubric Generation, Review & Validation"];
    C & D -- Governed by --> E["Task Bank of Reusable Criteria"];
    E --> F["Computational Processes: Writer Agent Research & Reviewer Agent Verification"];
    F -- Failures escalate to --> G["Human Intervention"];
    G --> H["Key Findings/Outcomes"];
    H --> I{"Rubrics track expert scoring & improve coverage"};
    H --> J{"Scores agree with human grading & analyst preference"};
    H --> K["FinAutoRubric Benchmark shows headroom for later models"];

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes FinAutoRubric, an expert‑guided framework that generates, reviews and validates task‑specific rubrics using reusable expert guidance.

Source sentence
We introduce FinAutoRubric, an expert‑guided rubric generation framework.

Assumes 1

  • Reusable expert guidance can govern rubric checks, expected‑value research, verification and weighting as of a stated information cutoff. other
    Source sentence
    Our key insight is that reusable expert guidance can govern not only what a rubric checks but also how its expected values are researched, verified, and weighted, before any response exists and as of a stated information cutoff.

Data

  • FinAutoRubric Benchmark · in‑house analysts benchmark unknown
    Source sentence
    The released 100‑query FinAutoRubric Benchmark, built from in‑house analysts' key questions across 78 tasks and eight asset classes

Costs

Not applicable The paper reports no trading or execution result.

Finds 4

  • On three expert‑authored finance benchmarks, its rubrics track expert scoring as closely as the strongest evaluated generator. performance
    Source sentence
    On three expert‑authored finance benchmarks, its rubrics track expert scoring as closely as the strongest evaluated generator
  • Analysts prefer FinAutoRubric rubrics in a blind review compared to baselines. other
    Source sentence
    analysts prefer them in a blind review
  • FinAutoRubric rubrics take more time and cost than baselines, which use a single model call or short fixed pipeline. other
    Source sentence
    These rubrics take more time and cost than those of the baselines, which come from a single model call or a short fixed pipeline
  • Rubrics written by an earlier model generation still leave room for improvement by a later model. other
    Source sentence
    rubrics written by an earlier model generation still leave room to separate a later one

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 19 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (71)

  1. FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents (2026) — link
  2. FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality (2026) — link
  3. Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction (2026) — link
  4. When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysis (2026) — link
  5. BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents (2026) — link
  6. PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges (2026) — link
  7. FinTradeBench: A Financial Reasoning Benchmark for LLMs — → 2603.19225
  8. FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets (2026) — link
and 63 more
  1. Qworld: Question-Specific Evaluation Criteria for LLMs (2026) — link
  2. RubricBench: Aligning Model-Generated Rubrics with Human Standards (2026) — link
  3. Evaluating LLMs in Finance Requires Explicit Bias Consideration — → 2602.14233
  4. Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings (2026) — link
  5. JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks (2026) — link
  6. APEX-Agents (2026) — link
  7. FinRetrieval: A Benchmark for Financial Data Retrieval by AI Agents (2026) — link
  8. DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research (2025) — link
  9. PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning (2025) — link
  10. ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge (2025) — link
  11. FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs (2025) — link
  12. OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment (2025) — link
  13. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks (2025) — link
  14. FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning (2025) — link
  15. FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering (2025) — link
  16. Your AI, Not Your View: The Bias of LLMs in Investment Analysis — → 2507.20957
  17. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains (2025) — link
  18. Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (2025) — link
  19. Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks (2025) — link
  20. FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation (2025) — link
  21. EvalAgent: Discovering Implicit Evaluation Criteria from the Web (2025) — link
  22. RocketEval: Efficient Automated LLM Evaluation via Grading Checklist (2025) — link
  23. FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Models (2025) — link
  24. TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation (2024) — link
  25. FinanceBench: A New Benchmark for Financial Question Answering (2023) — link
  26. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction (2018) — link
  27. Accelerated Hierarchical Density Based Clustering (2017) — link
  28. Density-Based Clustering Based on Hierarchical Density Estimates (2013) — link
  29. C10 Identifies the material downside risks, presents a concrete counter-case to the conclusion, and states observable developments that would invalidate or materially weaken the view
  30. P7 Numerical reconciliation and internal consistency. Inputs, intermediate results, terminal values, narrative statements, and conclusions must reconcile without contradiction
  31. C08 Keeps the stated inputs, credit metrics, valuation calculations, units, periods, intermediate results, and terminal outputs numerically consistent with one another
  32. E10 (5pt) Calculates LTM Gross Leverage as of September 30, 2025 as 7.45x, when rounded to two decimal places
  33. R4 (5pt) Reports consolidated QVC Group Adjusted OIBDA for the prior-year nine-month period (nine months ended September 30, 2024) that is removed in the LTM roll-forward
  34. E8 (1pt) Identify QVC cash and cash equivalents as of September 30, 2025 as $1,817.00M
  35. E13 Cash-adjusted EBITDA margin (after removing the $109 million equity award compensation add-back) was 24.20%
  36. E06 † States that Tesla, Inc
  37. C06 Attributes load-bearing disclosures, ratings, market observations, and comparable-credit inputs to identifiable sources and labels estimates, proxies, reconstructions, and unavailable inputs
  38. R9 (12pt) Nested revenue presentation for Tesla, Inc. (TSLA), Q4 2024 (three months ended December 31, 2024)
  39. C03 Uses inputs aligned to the correct entity, fiscal period, metric definition, operating scope, information basis, currency, and unit
  40. E9 (2pt) Calculate LTM Gross Leverage as the total consolidated debt outstanding as of September 30, 2025 of $6615.00M divided by LTM adjusted OIBDA as of September 30, 2025 of $880.0MM
  41. weeks ended 4/1/2025 segment table confirms
  42. R10 (1.6pt) Reconciliation add-back: states impairment losses for the twelve months ended December 28, 2024 as $3,669 million (USD millions; accept a value within 0.5 million)
  43. C04 Assesses liquidity, debt maturities, capital structure, and the analyzed instrument’s priority or security position sufficiently to support the credit view
  44. − 0 . 063 Investor return/payout
  45. Review input, skeleton bound to candidate, contract, evidence, and review-policy hashes
  46. Finance agent benchmark v2
  47. C02 Identifies every load-bearing input with traceable source attribution and labels disclosed, stipulated assumption-derived, estimated, proxy, and unavailable values accurately
  48. C03 Presents financial performance and core credit metrics on the correct entity, consolidation, accounting currency, unit, and reporting-period basis
  49. C02 Assesses the subject’s business and industry profile and identifies the factors that materially affect its capacity to service debt
  50. C09 Commits to a supported credit or investment-value conclusion and explains its implication relative to the stated comparison baseline or current market pricing
  51. R8 (12pt) Most recent reported quarter for Tesla, Inc. (TSLA) available through the February 4, 2025 information cutoff.
  52. 0 . 028 Multiples valuation. Consistent enterprise/equity and trailing/forward bases; aligned pricing dates; comparable multiple definitions
  53. E7 Unrealized losses/(gains) on commodity hedges were $(19) million
  54. passed FY ended 12/31/2024 segment table confirms Bubba’s 33 capital expenditures 38,557 and Total 354,341, matching the cash-flow line of (354,341) (F003,
  55. P6 Method, assumptions, and limitations transparency
  56. Feedback level, filtered review, and for revisions 2 and 3 the post-review scope report
  57. C04 States the unit-economics formula and applies the request’s allocation, counterfactual, cost, fee, or retention treatment to the shown inputs through reproducible intermediate calculations
  58. E12 Company-reported Adjusted EBITDA margin was 24.62%
  59. C01 Uses the requested issuer, obligor, instrument, and transaction terms as the subject of the credit analysis without substituting a different entity or security
  60. C01 Reports every requested operating-efficiency or unit-economics measure and comparison for the specified entity, operating scope, and period in the requested unit and form
  61. C07 Aligns financial evidence to the relevant reporting periods and gives an explicit as-of date or observation window for market-sensitive inputs and ratings
  62. E12 (5pt) Calculates Total Net Debt Outstanding as of September 30, 2025 as $4798.0M
  63. R14 (20pt) Identifies the equity award compensation expense separately excluded under Kraft Heinz’s Adjusted EBITDA definition as $109 million for fiscal 2024 (accept a value within 0.5 million),

3 of 71 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.