Paper: arXiv 2609.19866
Authors: Veronika Batzdorfer, Carlo Romano Marcello Alessandro Santagiustina
Abstract
High annotation reproducibility does not necessarily imply that an LLM-inferred measure captures the construct it is intended to measure. We test this distinction using a dataset from the European Commission’s AI Act consultation, linking structured survey responses to free-text consultation submissions from the same stakeholders. LLM annotations of consultation submissions are highly reproducible (intraclass correlations > 0.99), yet show limited convergence with survey-reported measures of the nominal construct they were intended to approximate. Divergence between survey-and LLM-inferred text-based measures varies systematically across stakeholder groups: business associations express greater concern about AI risks in text-based consultations than in survey responses ({g} = +1.0), whereas public authorities and several nonbusiness groups show smaller or negative divergences. Divergences between scores suggest positive spatial autocorrelation across European countries (Moran’s I = 0.347, p = 0.036), indicating that stakeholders from neighboring countries tend toward more similar text-based stances towards AI safety concerns. Despite divergence, survey-reported concerns remain strongly associated with support for explainability across all divergence levels. These results demonstrate that LLM annotation reproducibility can coexist with poor construct correspondence and motivate validation procedures that distinguish reproducibility, construct validity, and communication context variation when LLMs are used as measurement instruments.
Complexity vs Empirical Score
- Math Complexity: 3.0/10
- Empirical Rigor: 7.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: This paper presents a novel and empirically rigorous analysis of LLM measurement validity, distinguishing it from reproducibility. While the mathematical complexity is moderate, the robust statistical methodology and clear findings contribute to a high overall score. The insights into institutional communication are particularly valuable.
Research Flowchart
flowchart TD
A["Research Question: Reproducibility vs. Construct Validity in LLM Measurement?"] --> B{"Key Methodology: Comparing LLM annotations vs. Survey data"};
B --> C1["Data Input: EC AI Act Consultation Survey Responses"];
B --> C2["Data Input: EC AI Act Consultation Free-Text Submissions"];
C1 & C2 --> D["Computational Process: LLM Annotation & Statistical Analysis"];
D --> E{"Key Finding 1: High LLM Reproducibility (>0.99 ICC) but Poor Construct Convergence"};
D --> F{"Key Finding 2: Divergence Varies by Stakeholder (e.g., Business Associations: LLM > Survey AI Risk Concern, g=+1.0)"};
D --> G{"Key Finding 3: Spatial Autocorrelation of Divergence (Moran's I = 0.347) & Survey Concerns vs. Explainability Support"};
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper tests the distinction between annotation reproducibility and construct validity using linked survey and free‑text consultation data, and motivates validation procedures that separate reproducibility, construct validity, and communication context variation when LLMs are used as measurement
Source sentence
High annotation reproducibility does not necessarily imply that an LLM-inferred measure captures the construct it is intended to measure. These results demonstrate that LLM annotation reproducibility can coexist with poor construct correspondence and motivate validation procedures that distinguish reproducibility, construct validity, and communication context variation when LLMs are used as measurement instruments.
Assumes 1
- A reproducible LLM score can serve as a proxy for an underlying social construct when the construct is expressed through institutionally situated language. model
Source sentence
The design therefore provides a direct test of a central assumption in text-as-data research: that a reproducible LLM score can serve as a proxy for an underlying social construct when the construct is expressed through institutionally situated language.
Data
- European Commission AI Act consultation submissions text-news public
Source sentence
3 Material and methods 3.1 Data We use public consultations on the proposed European AI Regulation 2020, released via the Commission’s Better Regulation Portal4 and web-crawled from there. The dataset contains N = 857 submissions from 20-02-2020 – 27-04-2021 from 3 rounds.
- Linked structured survey responses measuring safety concern, rights concern, and explainab text-news public
Source sentence
Of these, N = 348 (40.6%) contain both free-text responses and structured survey items measuring safety concern, rights concern, and perceived explainability importance on a 0–5 scale.
Costs
Not applicable
Finds 5
- LLM annotations of consultation submissions are highly reproducible, with ICC(C,1) ranging from 0.994 to 0.996 and ICC(C,k) exceeding 0.999 for all constructs. statistical
Source sentence
To assess inter-run LLM annotation alignment, we ran five independent annotation passes. Inter-run agreement was exceptionally high: ICC(C,1) ranged from 0.994 to 0.996, and ICC(C,k) exceeded 0.999 for all constructs, indicating that the LLM was internally consistent—but consistency is not validity.5 3.3 Validation framework Before using LLM annotations as proxies for survey constructs, we examine four diagnostic
- LLM‑derived text measures show limited convergence with survey‑reported measures of the nominal construct they were intended to approximate. statistical
Source sentence
LLM annotations of consultation submissions are highly reproducible (intraclass correlations > 0.99), yet show limited convergence with survey-reported measures of the nominal construct they were intended to approximate.
- Divergence between text‑based and survey‑based scores varies systematically: business associations exhibit a positive divergence of +1.0, while public authorities show a negative divergence of –0.70. statistical figure from elsewhere in the paper
Source sentence
Divergence between survey-and LLM-inferred text-based measures varies systematically across stakeholder groups: business associations express greater concern about AI risks in text-based consultations than in survey responses ({g} = +1.0), whereas public authorities and several nonbusiness groups show smaller or negative divergences. Business associations show the largest positive divergence (¯g = +1.00, 95% CI
- Text‑survey divergence scores display positive spatial autocorrelation across European countries (Moran's I = 0.347, p = 0.036). statistical
Source sentence
Divergences between scores suggest positive spatial autocorrelation across European countries (Moran's I = 0.347, p = 0.036), indicating that stakeholders from neighboring countries tend toward more similar text-based stances towards AI safety concerns.
- Survey‑reported safety concerns remain strongly associated with support for explainability across all levels of text‑survey divergence. statistical
Source sentence
Despite divergence, survey-reported concerns remain strongly associated with support for explainability across all divergence levels.
Checked by others
No other paper in the archive has been found to support or contradict these findings. Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.