{"paper_id":"2410.05297","title":"Cyber Risk Taxonomies: Statistical Analysis of Cybersecurity Risk Classifications","authors":["Matteo Malavasi","Gareth W. Peters","Stefan Treuck","Pavel V. Shevchenko","Jiwook Jang","Georgy Sofronov"],"paper_date":"2024-10-04","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2410.05297v1","page_url":"https://thequant.space/flowcharts/cyber-risk-taxonomies-statistical-analysis-of-cybersecurity/","scores":{"empirical_rigor":8,"math_complexity":7,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes using a rolling window analysis with threshold weighted scoring functions to evaluate out‑of‑sample cyber risk forecasts.","quote":"We use a rolling window analysis, to compare cyber risk distribution forecasts via threshold weighted scoring functions.","summary":"Cyber risk classifications are widely used in the modeling of cyber event distributions, yet their effectiveness in out of sample forecasting performance remains underexplored. In this paper, we analyse the most commonly used classifications and argue in favour of switching the attention from…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"A high enough threshold u must be selected so exceedances follow a generalized Pareto distribution.","type":"statistical","stated":"explicit","quote":"First, the high enough threshold u must be selected such that the distribution of the corresponding exceedances converges to a generalized Pareto distribution."},{"assumption":"Bootstrap methodology is used to choose the lowest threshold that passes a GP exceedance test.","type":"statistical","stated":"explicit","quote":"(2009), where the appropriate threshold is selected as the lowest possible value such that a test of hypothesis of generalized Pareto distributed exceedances"},{"assumption":"Scoring rules (rCRPS, rES) rely on standardized residuals from a GAMLSS approach and can be applied even when the severity distribution lacks finite moments.","type":"model","stated":"explicit","quote":"The rCRPS and rES combine scoring rules, with the standardized residuals typical of the GAMLSS approach and allow the testing procedure to be carried out even in the case of a cyber event severity distribution without finite moments."},{"assumption":"The power analysis assumes the Type & Importance classification is the best among those considered.","type":"model","stated":"explicit","quote":"To ensure relevance to cyber risk, we consider the cyber event severity model based on the Type & Importance classification, as it is the classification returning arguably the best results among the considered classifications."},{"assumption":"Null hypothesis: out‑of‑sample performance of the Type & Importance classification is indistinguishable from no‑classification or random classification models.","type":"statistical","stated":"explicit","quote":"under the null hypothesis the out of sample performance of the Type & Importance classification is not distinguishable from the out of sample performance of the no classification model, or a random classification model."}]},"datasets":{"status":"available","items":[{"name":"Cyber event data (2009‑2014 estimation sample)","provider":null,"kind":"alternative","access":"unknown","period":"2009‑2014","universe":null,"quote":"We use the model estimated during the subsample period 2009 to 2014 to generate observations corresponding to the year 2015."},{"name":"Out‑of‑sample cyber events (2015)","provider":null,"kind":"alternative","access":"unknown","period":"2015","universe":null,"quote":"The total number of sample observations is 148 in the year 2015, from which the simulated universe of cyber risk losses consists of a total of 148,000 observations."},{"name":"Out‑of‑sample observations overall","provider":null,"kind":"alternative","access":"unknown","period":"nine years","universe":null,"quote":"in our out of sample study there are a total of 1,380 observations, collected over a period of nine years."},{"name":"Simulated universe of cyber risk losses","provider":null,"kind":"synthetic","access":"unknown","period":null,"universe":null,"quote":"the simulated universe of cyber risk losses consists of a total of 148,000 observations."}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"unclear"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"Cyber risk classifications yield unsatisfactory out‑of‑sample severity forecasts, often indistinguishable from random or no classification models.","kind":"negative","quote":"First, we find that cyber event severity models, estimated based on cyber risk classifications have an overall unsatisfactory forecasting performance, often not statistically distinguishable from a model with a random classifications or from a model without any classification."},{"finding":"Cyber risk classifications are useful for modeling event frequency but not severity.","kind":"performance","quote":"Second, while cyber risk classifications may be disregarded in severity modeling, they appear to be a valuable tool for modeling the frequency of cyber events."},{"finding":"Business‑motivated classifications are too restrictive and fail to capture heterogeneity of cyber events.","kind":"negative","quote":"Our results indicate that business motivated cyber risk classifications appear to be too restrictive and not flexible enough to capture the heterogeneity of cyber risk events."},{"finding":"Dynamic and impact‑based classifiers outperform other classifications for forecasting cyber loss severity.","kind":"performance","quote":"dynamic and impact-based cyber risk classifiers seem to be better suited in forecasting future cyber risk losses than the other considered classifications."},{"finding":"Power analysis shows the test rarely rejects the null with current sample sizes, implying larger samples are needed.","kind":"statistical","quote":"the null hypothesis is never rejected, indicating that the sample size required for the test statistic to correctly distinguish between the forecasting performances of the two models is far greater than the sample size considered."}],"reported_performance":null,"out_of_sample":{"method":"walk-forward","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":26,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T18:00:51Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}