Paper: arXiv 2406.15867

Authors: Thomas Cook, Patrick Flaherty

Abstract

Experimentation involves risk. The investigator expends time and money in the pursuit of data that supports a hypothesis. In the end, the investigator may find that all of these costs were for naught and the data fail to reject the null. Furthermore, the investigator may not be able to test other hypotheses with the same data set in order to avoid false positives due to p-hacking. Therefore, there is a need for a mechanism for investigators to hedge the risk of financial and statistical bankruptcy in the business of experimentation. In this work, we build on the game-theoretic statistics framework to enable an investigator to hedge their bets against the null hypothesis and thus avoid ruin. First, we describe a method by which the investigator’s test martingale wealth process can be capitalized by solving for the risk-neutral price. Then, we show that a portfolio that comprises the risky test martingale and a risk-free process is still a test martingale which enables the investigator to select a particular risk-return position using Markowitz portfolio theory. Finally, we show that a function that is derivative of the test martingale process can be constructed and used as a hedging instrument by the investigator or as a speculative instrument by a risk-seeking investor who wants to participate in the potential returns of the uncertain experiment wealth process. Together, these instruments enable an investigator to hedge the risk of ruin and they enable a investigator to efficiently hedge experimental risk.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 3.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper introduces an innovative concept bridging game-theoretic statistics and financial derivative pricing to hedge hypothesis testing risk. While mathematically rigorous and conceptually novel, it relies heavily on theoretical derivations and basic toy simulations rather than extensive empirical backtesting with real-world statistical experimental data.

Research Flowchart

  flowchart TD
    A["Research Goal: Hedge financial & statistical risk in sequential experiments"] --> B["Base Input: Test Martingale Wealth Process"]
    B --> C["Step 1: Calculate Risk-Neutral Price to Capitalize Test Martingale"]
    C --> D["Step 2: Combine with Risk-Free Asset via Markowitz Portfolio Theory"]
    D --> E["Step 3: Construct Derivative Contracts on the Test Martingale"]
    E --> F["Outcomes: Avoid Experimenter Ruin & Enable Risk-Seeking Speculation"]

Cited References (114)

  1. Efficient Adaptive Experimental Design for Average Treatment Effect Estimation (2021) — link
  2. Statistical Inference with M -Estimators on Adaptively Collected Data (2021)
  3. Asymptotic normality for sums of dependent random variables (1972)
  4. Anytime-valid off-policy inference for contextual bandits (2024)
  5. Time-uniform, nonparametric, nonasymptotic confidence sequences (2021)
  6. Time-uniform central limit theory and asymptotic confidence sequences (2023) — link
  7. Estimating means of bounded random variables by betting (2023)
  8. Game-theoretic statistics and safe anytime-valid inference (2023) — link
and 106 more
  1. Adaptive Experimental Design Using the Propensity Score (2011)
  2. Experiment-as-Market: Incorporating Welfare into Randomized Controlled Trials (2018)
  3. Always Valid Inference: Continuous Monitoring of A/B Tests (2022)
  4. Sharp Finite-Time Iterated-Logarithm Martingale Concentration (2015) — link
  5. Sequential Nonparametric Testing with the Law of the Iterated Logarithm (2016)
  6. The Adaptive Doubly Robust Estimator and a Paradox Concerning Logging Policy (2021)
  7. Adaptive Doubly Robust Estimator from Non-stationary Logging Policy under a Convergence of Average Probability (2021) — link
  8. Some aspects of the sequential design of experiments (1952)
  9. Estimation of Regression Coefficients When Some Regressors Are Not Always Observed (1994)
  10. Randomization Analysis of Experimental Data: The Fisher Randomization Test Comment (1980)
  11. Comment: Which Ifs Have Causal Answers (1986)
  12. Double/debiased machine learning for treatment and structural parameters (2018)
  13. Statistical Methods Related to the Law of the Iterated Logarithm (1970)
  14. Confidence Sequences for Mean, Variance, and Median (1967)
  15. Multi-armed Bandit Experimental Design: Online Decision-making and Adaptive Inference (2023)
  16. On the near-optimality of betting confidence sets for bounded means (2023) — link
  17. Peeking at A / B Tests : Why It Matters, and What to Do about It (2017)
  18. CLIP - OGD : An Experimental Design for Adaptive Neyman Allocation in Sequential Experiments (2023)
  19. Asymptotic Efficiency Bounds for a Class of Experimental Designs (2022) — link
  20. Confidence intervals for policy evaluation in adaptive experiments (2021)
  21. CONOPT—a large-scale GRG code (1994)
  22. Controlling the false discovery rate: a practical and powerful approach to multiple testing (1995)
  23. The philosophy of multiple comparisons (1991)
  24. The Collected Works of John W. Tukey: Multiple Comparions (1994)
  25. $\alpha$-investing: a procedure for sequential control of expected false discoveries (2008)
  26. Generalized $\alpha$-investing: definitions, optimality results and application to public databases (2014)
  27. Online control of the false discovery rate with decaying memory (2017)
  28. Asynchronous Online Testing of Multiple Hypotheses. (2021)
  29. Online rules for control of false discovery rate and false discovery exceedance (2018)
  30. SAFFRON: an adaptive algorithm for online control of the false discovery rate (2018)
  31. A direct approach to false discovery rates (2002)
  32. Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach (2004)
  33. A framework for multi-a (rmed)/b (andit) testing with online fdr control (2017)
  34. A sequential algorithm for false discovery rate control on directed acyclic graphs (2019)
  35. Sequential Bonferroni methods for multiple hypothesis testing with strong control of family-wise error rates I and II (2012)
  36. Adaptive and dynamic adaptive procedures for false discovery rate control and estimation (2012)
  37. Adaptive linear step-up procedures that control the false discovery rate (2006)
  38. FDR control with adaptive procedures and FDR monotonicity (2011)
  39. Gene expression correlates of clinical prostate cancer behavior (2002)
  40. BagBoosting for tumor classification with gene expression data (2004)
  41. Decision theory: Principles and approaches (2009)
  42. Theory of games and statistical decisions (1979)
  43. Bayes and minimax solutions of sequential decision problems (1949)
  44. The weighted likelihood ratio, sharp hypotheses about chances, the order of a Markov chain (1970)
  45. Computing Bayes factors using a generalization of the Savage-Dickey density ratio (1995)
  46. Statistical decision theory and Bayesian analysis (2013)
  47. Contextual online false discovery rate control (2020)
  48. Neural FDR : Learning discovery thresholds from hypothesis features (2017)
  49. Transforming L1000 profiles to RNA-seq-like profiles with deep learning. (2022)
  50. Streaming Feature Selection Using Alpha-Investing (2005) — link
  51. Edgar R, Domrachev M, Lash AEGene Expression Omnibus: NCBI gene expression and hybridization array data repository. Nucl Acids Res 30: 207-210 (2002) — link
  52. The Power of Batching in Multiple Hypothesis Testing (2020)
  53. Dynamic Algorithms for Online Multiple Testing (2022)
  54. Online Multiple Hypothesis Testing (2023) — link
  55. onlineFDR: an R package to control the false discovery rate for growing data repositories (2019) — link
  56. Online multiple testing with e-values (2023) — link
  57. Probability and finance: it’s only a game! (2005)
  58. Testing by Betting: A Strategy for Statistical and Scientific Communication (2021)
  59. Test Martingales, Bayes Factors and p-Values (2011)
  60. Multiple Range and Multiple F Tests (1955)
  61. THE DISTRIBUTION OF RANGE IN SAMPLES FROM A NORMAL POPULATION, EXPRESSED IN TERMS OF AN INDEPENDENT ESTIMATE OF STANDARD DEVIATION (1939) — link
  62. The use of the ``studentized range’’ in connection with an analysis of variance (1952) — link
  63. The Analysis of Variance (1959) — link
  64. Comparing Individual Means in the Analysis of Variance (1949)
  65. Applications of the Theory of Error to the Interpretation of Trends (1929) — link
  66. Multiple Comparisons Among Means (1961)
  67. A Simple Sequentially Rejective Multiple Test Procedure (1979)
  68. A sharper Bonferroni procedure for multiple tests of significance (1988) — link
  69. A Course In Game Theory (2020) — link
  70. Zur Theorie der Gesellschaftsspiele (1928) — link
  71. The Compleat Strategyst: Being a Primer on the Theory of Games of Strategy (1986) — link
  72. Teoria statistica delle classi e calcolo delle probabilit `a (1936) — link
  73. The Mathematical Analysis of Logic (1847) — link
  74. On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9. (1923)
  75. Statistical Methods for Research Workers (1925) — link
  76. On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection (1934)
  77. On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects (1998)
  78. Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score (2003) — link
  79. The testing of statistical hypotheses in relation to probabilities a priori (1933)
  80. ON THE USE AND INTERPRETATION OF CERTAIN TEST CRITERIA FOR PURPOSES OF STATISTICAL INFERENCE PART I (1928) — link
  81. On Confidence Sequences (1976) — link
  82. Time-uniform Chernoff bounds via nonnegative supermartingales (2018) — link
  83. Admissible anytime-valid sequential inference must rely on nonnegative martingales (2022) — link
  84. Anytime-Valid Confidence Sequences in an Enterprise A/B Testing Platform (2023) — link
  85. Sequential Tests of Statistical Hypotheses (1945) — link
  86. Sequential Analysis (1947)
  87. Safe Testing (2024)
  88. E-values: Calibration, combination and applications (2021) — link
  89. Game-Theoretic Foundations for Probability and Finance (2019) — link
  90. Probability Theory (1977)
  91. Investment Science (2014) — link
  92. Auditing Fairness by Betting (2023) — link
  93. Théorie de la spéculation (1900) — link
  94. Rational Theory of Warrant Pricing (1965) — link
  95. The Pricing of Options and Corporate Liabilities (1973)
  96. Portfolio Selection (1952)
  97. Tight Concentrations and Confidence Sequences From the Regret of Universal Portfolio (2024) — link
  98. Statistical Decision Functions (1950)
  99. Sequential Estimation (1953) — link
  100. Asymptotically subminimax solutions of compound statistical decision problems, 1951 (1951)
  101. Stochastic Processes (1953)
  102. The Doctrine of Chances: Probabilistic Aspects of Gambling (2010)
  103. A New Interpretation of Information Rate (1956) — link
  104. Solving Stochastic Programming Problems with Risk Measures by Progressive Hedging (2018) — link
  105. Étude critique de la notion de collectif (1939)
  106. Cost-Aware Generalized $\alpha$-Investing for Multiple Hypothesis Testing (2024) — link

0 of 114 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.