{"paper_id":"2609.19866","title":"Reproducibility is not construct validity: LLM measurement of institutionally situated communication","authors":["Veronika Batzdorfer","Carlo Romano Marcello Alessandro Santagiustina"],"paper_date":"2026-09-17","paper_type":"survey-review","paper_url":"https://arxiv.org/pdf/2609.19866v1","page_url":"https://thequant.space/flowcharts/reproducibility-is-not-construct-validity-llm-measurement-of-institutionally/","scores":{"empirical_rigor":7,"math_complexity":3,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper tests the distinction between annotation reproducibility and construct validity using linked survey and free‑text consultation data, and motivates validation procedures that separate reproducibility, construct validity, and communication context variation when LLMs are used as measurement ","quote":"High annotation reproducibility does not necessarily imply that an LLM-inferred measure captures the construct it is intended to measure. These results demonstrate that LLM annotation reproducibility can coexist with poor construct correspondence and motivate validation procedures that distinguish reproducibility, construct validity, and communication context variation when LLMs are used as measurement instruments.","summary":"High annotation reproducibility does not necessarily imply that an LLM-inferred measure captures the construct it is intended to measure. We test this distinction using a dataset from the European Commission's AI Act consultation, linking structured survey responses to free-text consultation…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"A reproducible LLM score can serve as a proxy for an underlying social construct when the construct is expressed through institutionally situated language.","type":"model","stated":"explicit","quote":"The design therefore provides a direct test of a central assumption in text-as-data research: that a reproducible LLM score can serve as a proxy for an underlying social construct when the construct is expressed through institutionally situated language.","support":"full"}]},"datasets":{"status":"available","items":[{"name":"European Commission AI Act consultation submissions","provider":"European Commission","kind":"text-news","access":"public","period":"2020-04-20 to 2021-04-27","universe":null,"quote":"3 Material and methods 3.1 Data We use public consultations on the proposed European AI Regulation 2020, released via the Commission’s Better Regulation Portal4 and web-crawled from there. The dataset contains N = 857 submissions from 20-02-2020 – 27-04-2021 from 3 rounds.","support":"full"},{"name":"Linked structured survey responses measuring safety concern, rights concern, and explainab","provider":"European Commission","kind":"text-news","access":"public","period":"2020-04-20 to 2021-04-27","universe":null,"quote":"Of these, N = 348 (40.6%) contain both free-text responses and structured survey items measuring safety concern, rights concern, and perceived explainability importance on a 0–5 scale.","support":"partial"}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"none"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"none","findings":[{"finding":"LLM annotations of consultation submissions are highly reproducible, with ICC(C,1) ranging from 0.994 to 0.996 and ICC(C,k) exceeding 0.999 for all constructs.","kind":"statistical","quote":"To assess inter-run LLM annotation alignment, we ran five independent annotation passes. Inter-run agreement was exceptionally high: ICC(C,1) ranged from 0.994 to 0.996, and ICC(C,k) exceeded 0.999 for all constructs, indicating that the LLM was internally consistent—but consistency is not validity.5 3.3 Validation framework Before using LLM annotations as proxies for survey constructs, we examine four diagnostic","support":"full"},{"finding":"LLM‑derived text measures show limited convergence with survey‑reported measures of the nominal construct they were intended to approximate.","kind":"statistical","quote":"LLM annotations of consultation submissions are highly reproducible (intraclass correlations > 0.99), yet show limited convergence with survey-reported measures of the nominal construct they were intended to approximate.","support":"full"},{"finding":"Divergence between text‑based and survey‑based scores varies systematically: business associations exhibit a positive divergence of +1.0, while public authorities show a negative divergence of –0.70.","kind":"statistical","quote":"Divergence between survey-and LLM-inferred text-based measures varies systematically across stakeholder groups: business associations express greater concern about AI risks in text-based consultations than in survey responses ({g} = +1.0), whereas public authorities and several nonbusiness groups show smaller or negative divergences. Business associations show the largest positive divergence (¯g = +1.00, 95% CI","support":"partial"},{"finding":"Text‑survey divergence scores display positive spatial autocorrelation across European countries (Moran's I = 0.347, p = 0.036).","kind":"statistical","quote":"Divergences between scores suggest positive spatial autocorrelation across European countries (Moran's I = 0.347, p = 0.036), indicating that stakeholders from neighboring countries tend toward more similar text-based stances towards AI safety concerns.","support":"full"},{"finding":"Survey‑reported safety concerns remain strongly associated with support for explainability across all levels of text‑survey divergence.","kind":"statistical","quote":"Despite divergence, survey-reported concerns remain strongly associated with support for explainability across all divergence levels.","support":"full"}],"reported_performance":null,"out_of_sample":{"method":"none","headline_out_of_sample":null}},"independent_evidence":{"status":"not_extracted","supports":[],"contradicts":[],"qualifies":[],"compared_with":0,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":2,"extracted_at":"2026-11-14T19:39:34Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}