How Well Do LLMs Perform in Peer Review? Evidence from Finance and Accounting

Paper: arXiv 2610.18435

Authors: Hai Lu, Yichen Luo, Luping Yu

Abstract

Referees increasingly use large language models (LLMs), while finance and accounting journals disagree about whether to allow it. We ask which part of a referee’s job the models can do. GPT-4o, whose training data end before the 2024 conference season, referees the papers presented at the 2024 AFA and NBER finance meetings, and we compare its verdicts with publication in the top-three finance journals over the next two years. Papers the model scores one standard deviation higher are 44% more likely to be published, net of author reputation. The result is not driven by look-ahead bias, as the model does not recall the outcomes. The information comes from its judgements of correctness and exposition. Its judgements of originality and significance carry none, and a referee guide that stresses importance does not help. The split between correctness and contribution holds for accounting papers and for a newer model, GPT-5. LLMs can check whether research is right, but not whether it matters. The technical check can go to the model, while the judgement of contribution stays with people.

Complexity vs Empirical Score

• Math Complexity: 3.0/10
• Empirical Rigor: 8.0/10
• Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: This paper presents a highly novel and empirically rigorous study on LLMs in peer review, demonstrating their capability in assessing correctness but not contribution. The methodology is clear and the findings are significant for academic publishing. While the math complexity is low, the strong empirical design and novel application make it a valuable contribution.

Research Flowchart

  flowchart TD
    A[Research Goal/Question: How well do LLMs perform in peer review?] --> B{Methodology: LLM Refereeing Simulation};
    B --> C[Data/Inputs: GPT-4o referees 2024 AFA/NBER papers];
    C --> D{Computational Process: Compare GPT-4o scores with Top-3 finance journal publication};
    D --> E[Key Findings/Outcomes: LLMs check correctness but not significance; technical check can be automated];