Paper: arXiv 2410.05297
Authors: Matteo Malavasi, Gareth W. Peters, Stefan Treuck, Pavel V. Shevchenko, Jiwook Jang, Georgy Sofronov
Abstract
Cyber risk classifications are widely used in the modeling of cyber event distributions, yet their effectiveness in out of sample forecasting performance remains underexplored. In this paper, we analyse the most commonly used classifications and argue in favour of switching the attention from goodness-of-fit and in-sample predictive performance, to focusing on the out-of sample forecasting performance. We use a rolling window analysis, to compare cyber risk distribution forecasts via threshold weighted scoring functions. Our results indicate that business motivated cyber risk classifications appear to be too restrictive and not flexible enough to capture the heterogeneity of cyber risk events. We investigate how dynamic and impact-based cyber risk classifiers seem to be better suited in forecasting future cyber risk losses than the other considered classifications. These findings suggest that cyber risk types provide limited forecasting ability concerning cyber event severity distribution, and cyber insurance ratemakers should utilize cyber risk types only when modeling the cyber event frequency distribution. Our study offers valuable insights for decision-makers and policymakers alike, contributing to the advancement of scientific knowledge in the field of cyber risk management.
Complexity vs Empirical Score
- Math Complexity: 7.0/10
- Empirical Rigor: 8.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper employs advanced statistical methods including dynamic extreme value theory, GAMLSS, and sophisticated scoring rules (CRPS, ES, rCRPS, rES), indicating high mathematical complexity. It also demonstrates strong empirical rigor through the use of a real-world dataset (Advisen), rolling-window out-of-sample forecasting, and simulation studies to validate findings.
Research Flowchart
flowchart TD
A["Research Goal: Assess Cyber Risk Classifications for Out-of-sample Forecasting"] --> B{"Data & Methodology"}
B --> C["Input: Cyber Event Data<br>Risk Classifications<br>Threshold Weighted Scoring"]
C --> D["Computational Process:<br>Rolling Window Analysis"]
D --> E["Evaluate Out-of-sample<br>Distribution Forecasts"]
E --> F["Key Findings & Outcomes"]
F --> G["Dynamic & Impact-based<br>Classifiers perform best"]
F --> H["Business-motivated<br>Classifiers too restrictive"]
F --> I["Recommendation:<br>Use types for Frequency,<br>not Severity distribution"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes using a rolling window analysis with threshold weighted scoring functions to evaluate out‑of‑sample cyber risk forecasts.
Source sentence
We use a rolling window analysis, to compare cyber risk distribution forecasts via threshold weighted scoring functions.
Assumes 5
- A high enough threshold u must be selected so exceedances follow a generalized Pareto distribution. statistical
Source sentence
First, the high enough threshold u must be selected such that the distribution of the corresponding exceedances converges to a generalized Pareto distribution.
- Bootstrap methodology is used to choose the lowest threshold that passes a GP exceedance test. statistical
Source sentence
(2009), where the appropriate threshold is selected as the lowest possible value such that a test of hypothesis of generalized Pareto distributed exceedances
- Scoring rules (rCRPS, rES) rely on standardized residuals from a GAMLSS approach and can be applied even when the severity distribution lacks finite moments. model
Source sentence
The rCRPS and rES combine scoring rules, with the standardized residuals typical of the GAMLSS approach and allow the testing procedure to be carried out even in the case of a cyber event severity distribution without finite moments.
- The power analysis assumes the Type & Importance classification is the best among those considered. model
Source sentence
To ensure relevance to cyber risk, we consider the cyber event severity model based on the Type & Importance classification, as it is the classification returning arguably the best results among the considered classifications.
Show 1 more
- Null hypothesis: out‑of‑sample performance of the Type & Importance classification is indistinguishable from no‑classification or random classification models. statistical
Source sentence
under the null hypothesis the out of sample performance of the Type & Importance classification is not distinguishable from the out of sample performance of the no classification model, or a random classification model.
Data
- Cyber event data (2009‑2014 estimation sample) alternative unknown
Source sentence
We use the model estimated during the subsample period 2009 to 2014 to generate observations corresponding to the year 2015.
- Out‑of‑sample cyber events (2015) alternative unknown
Source sentence
The total number of sample observations is 148 in the year 2015, from which the simulated universe of cyber risk losses consists of a total of 148,000 observations.
- Out‑of‑sample observations overall alternative unknown
Source sentence
in our out of sample study there are a total of 1,380 observations, collected over a period of nine years.
- Simulated universe of cyber risk losses synthetic unknown
Source sentence
the simulated universe of cyber risk losses consists of a total of 148,000 observations.
Costs
Not applicable
Finds 5
- Cyber risk classifications yield unsatisfactory out‑of‑sample severity forecasts, often indistinguishable from random or no classification models. negative
Source sentence
First, we find that cyber event severity models, estimated based on cyber risk classifications have an overall unsatisfactory forecasting performance, often not statistically distinguishable from a model with a random classifications or from a model without any classification.
- Cyber risk classifications are useful for modeling event frequency but not severity. performance
Source sentence
Second, while cyber risk classifications may be disregarded in severity modeling, they appear to be a valuable tool for modeling the frequency of cyber events.
- Business‑motivated classifications are too restrictive and fail to capture heterogeneity of cyber events. negative
Source sentence
Our results indicate that business motivated cyber risk classifications appear to be too restrictive and not flexible enough to capture the heterogeneity of cyber risk events.
- Dynamic and impact‑based classifiers outperform other classifications for forecasting cyber loss severity. performance
Source sentence
dynamic and impact-based cyber risk classifiers seem to be better suited in forecasting future cyber risk losses than the other considered classifications.
- Power analysis shows the test rarely rejects the null with current sample sizes, implying larger samples are needed. statistical
Source sentence
the null hypothesis is never rejected, indicating that the sample size required for the test statistic to correctly distinguish between the forecasting performances of the two models is far greater than the sample size considered.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 26 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (98)
- The drivers of cyber risk (2020)
- Insurability of cyber risk: An empirical analysis (2015)
- Modelling and management of cyber risk (2015)
- Insurability of cyber risk: An empirical analysis (2015)
- Content analysis of cyber insurance policies: How do carriers price cyber risk? (2019)
- Cyber risk and the changing role of insurance (2017)
- Impact of insurance for operational risk: Is it worthwhile to insure or be insured for severe losses? (2011)
- Cyber risk for the financial sector: A framework for quantitative assessment (2018)
and 90 more
- Cyber-risk decision models: To insure IT or not? (2013)
- Cyber-risk management (2015)
- Cyber risk research in business and actuarial science (2020)
- What do we know about cyber risk and cyber risk insurance? (2016)
- Data breaches: Goodness of fit, pricing, and risk measurement (2017)
- Cyberattacks on critical infrastructure: An economic perspective (2019)
- Residual life time at great age (1974)
- An extreme value approach for modeling operational risk losses depending on covariates (2016)
- A taxonomy of operational cyber security risks (2010)
- A taxonomy of operational cyber security risks version 2 (2014)
- Operational loss scaling by exposure indicators: evidence from the ORX database (2008)
- A Clearer Vision for Assessing the Risk of Cyber Incidents (2020)
- Scaling models for the severity and frequency of external operational loss data (2010)
- What are the actual costs of cyber risk events? (2019)
- Data breaches: Goodness of fit, pricing, and risk measurement (2017)
- Hype and heavy tails: A closer look at data breaches (2016)
- Human behaviour as an aspect of cybersecurity assurance (2016)
- A scaling model for severity of operational losses using generalized additive models for location scale and shape (GAMLSS) (2013)
- Heavy-tailed distribution of cyber-risks (2010)
- Examining the costs and causes of cyber incidents (2016)
- Statistical inference using extreme order statistics (1975)
- Cost of Data Breach Report 2019 (2019)
- Likelihood ratio tests for model selection and non-nested hypotheses (1989)
- Cyber risk ordering with rank-based statistical models (2020)
- Dealing with the biased effects issue when handling huge datasets: the case of INVALSI data (2015)
- A simple sequentially rejective multiple test procedure (1979)
- Cyber risk ordering with rank-based statistical models (2020)
- Statistical machine learning analysis of cyber risk data: event case studies (2018)
- Understanding cyber-risk and cyber-insurance (2018)
- Generalized additive models for location, scale and shape (2005)
- Generalized additive models for location scale and shape ( GAMLSS ) in R (2008)
- Statistics of extremes by oracle estimation (2008)
- Nonparametric adaptive estimation of conditional probabilities of rare events and extreme quantiles (2015)
- A simple general approach to inference about the tail of a distribution (1975)
- Flexible regression and smoothing: using GAMLSS in R (2017)
- The modelling of operational risk: experience with the analysis of the data collected by the Basel Committee (2004)
- Capital and risk: New evidence on implications of large operational losses (2006)
- A tale of tails: an empirical analysis of loss distribution models for estimating operational risk capital (2006)
- Data clustering with actuarial applications (2020)
- An integrated system for estimating the risk premium of individual car models in motor insurance (1986)
- Model-based and nonparametric approaches to clustering for data compression in actuarial applications (2017)
- An empirical comparison of some experimental designs for the valuation of large variable annuity portfolios (2016)
- A data mining framework for valuing large portfolios of variable annuities (2017)
- Pattern recognition by an adaptive process of sample set construction (1962)
- Some methods for classification and analysis of multivariate observations (1967)
- A bootstrap goodness of fit test for the generalized Pareto distribution (2009)
- A simple method for testing goodness of fit in the presence of nuisance parameters (1980)
- The exact and asymptotic distributions of C ram 'e r-von M ises statistics (1996)
- A review of quadratic discriminant analysis for high-dimensional data (2018)
- Transforming big data into smart data: An insight on the use of the k-nearest neighbors algorithm to obtain quality data (2019)
- Application of K-nearest neighbors algorithm on breast cancer diagnosis problem. (2000)
- Extreme Data Breach Losses: An Alternative Approach to Estimating Probable Maximum Loss for Data Breach Risk (2021)
- Copula approaches for modeling cross-sectional dependence of data breach losses (2018)
- Problems with scoring methods and ordinal scales in risk assessment (2010)
- Attack chain detection (2015)
- The use of the rank transform in regression (1979)
- Infinite mean models and the LDA for operational risk (2006)
- The calculation of minimum regulatory capital using single-loss approximations (2010)
- Probabilistic forecasts, calibration and sharpness (2007)
- Comparing density forecasts using threshold-and quantile-weighted scoring rules (2011)
- Comparing density forecasts via weighted likelihood ratio tests (2007)
- Axiomatic characterization of the quadratic scoring rule (1998)
- Strictly proper scoring rules, prediction, and estimation (2007)
- Cyber loss model risk translates to premium mispricing and risk sensitivity (2023)
- Cyber risk frequency, severity and insurance viability (2022)
- Evaluating the discrimination ability of proper multi-variate scoring rules (2022)
- Energy statistics: A class of statistics based on distances (2013)
- New goodness-of-fit tests for Pareto distributions (2009)
- Comparing density forecasts via weighted likelihood ratio tests (2007)
- Standards for security categorization of federal information and information systems (2004)
- Australian Cyber Security Centre: Annual Cyber Threat Report. (2021)
- An investigation of cyber loss data and its links to operational risk (2019)
- Defining cyber risk (2021)
- The drivers of cyber risk (2022)
- Cyber risk management in the US banking and insurance industry: A textual and empirical analysis of determinants and value (2022)
- Reducing informational disadvantages to improve cyber risk management (2018)
- Cyber risk management: History and future research directions (2021)
- Cyber risk and the US financial system: A pre-mortem analysis (2022)
- The future of financial stability and cyber risk (2018)
- Cyber risk definition and classification for financial risk management (2023)
- Cyber runs: How a cyber attack could affect US financial institutions (2019)
- Some principles for regulating cyber risk (2019)
- Cyber risk and financial stability: It’sa small world after all (2020)
- Managing cyber risk, a science in the making (2023)
- Building up cyber resilience by better grasping cyber risk via a new algorithm for modelling heavy-tailed data (2023)
- Cyber claim analysis using Generalized Pareto regression trees with applications to insurance (2021)
- The nature of losses from cyber-related events: risk categories and business sectors (2023)
- European Cybersecurity Centres of Expertise Map - Definitions and Taxonomy (2018)
- A proposal for a European cybersecurity taxonomy (2019)
- CRO forum concept paper on a proposed categorisation methodology for cyber risk (2017)
0 of 98 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.