Paper: arXiv 2604.17593

Authors: Yoshimasa Uematsu, Shinya Tanaka

Abstract

We propose post-screening portfolio selection (PS$^2$), a two-step framework for high-dimensional mean–variance investing. First, assets are screened by Lasso-type regression of a constant on excess returns without an intercept. Second, portfolio weights are estimated on the selected set using standard low-dimensional methods. Because strong factors can destroy sparsity in real data, we further introduce PS$^2$ with factors (FPS$^2$), which defactors returns before screening and allows factor investing in the final step. We establish theoretical guarantees, and simulations and an empirical application show competitive performance, especially when sparse screening is appropriate or strong factors are explicitly accommodated.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper combines rigorous high-dimensional statistical theory and proofs (N/T -> infinity) with a concrete two-step empirical framework applied to S&P 500 data and Monte Carlo simulations.

Research Flowchart

  flowchart TD
    A[<b>Goal:</b> High-dimensional Mean-Variance Investing] --> B[<b>Input:</b> Excess Returns Data]
    B --> C{<b>Methodology</b>}
    C -->|Core| D[PS²: Lasso Screening & Low-dim Weighting]
    C -->|Advanced| E[FPS²: Defactoring & Factor Investing]
    D --> F[<b>Outcomes:</b> Theoretical Guarantees]
    E --> F
    F --> G[<b>Performance:</b> Superiority in Sparse/Factor-heavy Markets]

Cited References (223)

  1. Forecasting using a large number of predictors: Is B ayesian shrinkage a valid alternative to principal components? (2008)
  2. High-dimensional classification using features annealed independence rules (2008)
  3. Impacts of high dimensionality in finite samples (2013)
  4. Asymptotic equivalence of regularization methods in thresholded parameter space (2013)
  5. A general theory of hypothesis tests and confidence regions for sparse high dimensional models (2017)
  6. Sure independence screening for ultrahigh dimensional feature space (with discussion) (2008)
  7. Estimating False Discovery Proportion Under Arbitrary Covariance Dependence (with discussion) (2012)
  8. Inference in Approximately Sparse Correlated Random Effects Probit Models (2018)
and 215 more
  1. Confidence Bands for Coefficients in High Dimensional Linear Models with Error-in-Variables (2017) — link
  2. Double/Debiased Machine Learning for Treatment and Structural Parameters (2018)
  3. Double/De-Biased Machine Learning Using Regularized R iesz Representers (2018) — link
  4. Simultaneous Inference for Best Linear Predictor of the Conditional Average Treatment Effect and Other Structural Functions (2018)
  5. Goodness-of-Fit Tests for High Dimensional Linear Models (2018)
  6. Efficient Predictor Ranking and False Discovery Proportion Control in High-Dimensional Regression (2018) — link
  7. Simultaneous Inference for High-Dimensional Linear Models (2017)
  8. Inference for High-Dimensional Instrumental Variables Regression (2018) — link
  9. Two-Step Estimation and Inference with Possibly Many Included Covariates (2018) — link
  10. Linear Hypothesis Testing in Dense High-Dimensional Linear Models (2017)
  11. Confidence Intervals for Causal Effects with Invalid Instruments by Using Two-Stage Hard Thresholding with Voting (2018)
  12. Accuracy Assessment for High-Dimensional Linear Regression (2018)
  13. Inference in Linear Regression Models with Many Covariates and Heteroscedasticity (2018)
  14. Asymptotic Confidence Regions for High-Dimensional Structured Sparsity (2018)
  15. Algorithmic Learning in a RandomWorld (2005)
  16. Distribution-Free Predictive Inference for Regression (2018)
  17. Sparsity Oriented Importance Learning for High-Dimensional Linear Regression (2018)
  18. Selective Inference with a Randomized Response (2018)
  19. Uniform Inference in High-Dimensional Dynamic Panel Data Models with Approximately Sparse Fixed Effects (2019)
  20. High-Dimensional Panel Data with Time Heterogeneity: Estimation and Inference (2017)
  21. Inferences in Panel Data with Interactive Effects Using Large Covariance Matrices (2017)
  22. LASSO-Driven Inference in Time and Space (2018) — link
  23. Minimum Distance Approach to Inference with Many Instruments (2018)
  24. High-Dimensional Econometrics and Regularized GMM (2018) — link
  25. Random forests (2001)
  26. Probability and Measure (3rd ed.) (1995)
  27. Nonuniformity of p-values can occur early in diverging dimensions (2017) — link
  28. On asymptotically optimal confidence regions and tests for high-dimensional models (2014)
  29. Valid post-selection inference (2013)
  30. Confidence intervals and hypothesis testing for high-dimensional regression (2014)
  31. Exact post-selection inference, with application to the lasso (2016)
  32. Optimal inference after model selection (2014) — link
  33. Confidence intervals for low dimensional parameters in high dimensional linear models (2014)
  34. Model selection and inference: Facts and fiction (2005)
  35. Can one estimate the conditional distribution of post-model-selection estimators? (2006)
  36. Sparse estimators and the oracle property, or the return of Hodges ’ estimator (2008)
  37. On the distribution of penalized maximum likelihood estimators: The L A S S O , S C A D , and thresholding (2009)
  38. Concentration Inequalities for Sums and Martingales (1st ed.) (2015)
  39. Matrix Analysis (2nd ed.) (2012)
  40. Introduction to the Non-Asymptotic Analysis of Random Matrices (2012)
  41. Some Applications of the M alliavin calculus to Sub- G aussian and Non-Sub- G aussian Random Fields (2007)
  42. Probability: Theory and Examples (4th ed.) (2010)
  43. High Dimensional Statistics (2017)
  44. Eigenvalue ratio test for the number of factors (2013)
  45. RANK : large-scale inference with graphical nonlinear knockoffs (2017) — link
  46. A One Covariate at a Time, Multiple Testing Approach to Variable Selection in High-Dimensional Linear Regression Models (2018)
  47. Controlling the false discovery rate via knockoffs (2015)
  48. A knockoff filter for high-dimensional selective inference (2016) — link
  49. Consistency for a simple model of random forests (2004)
  50. Controlling the false discovery rate: a practical and powerful approach to multiple testing (1995)
  51. The control of the false discovery rate in multiple testing under dependency (2001)
  52. Discovering the false discovery rate (2010)
  53. Il calcolo delle assicurazioni su gruppi di teste (1935)
  54. A simple sequentially rejective multiple test procedure (1979)
  55. Exact and approximate stepdown methods for multiple hypothesis testing (2005)
  56. Panning for gold: `model-X’ knockoffs for high dimensional controlled variable selection (2018)
  57. Robust inference with knockoffs (2018) — link
  58. Regression shrinkage and selection via the lasso (1996)
  59. Generalized shrinkage methods for forecasting using many predictors (2012)
  60. Macroeconomic forecasting using diffusion indexes (2002)
  61. Determining the number of factors in approximate factor models (2002)
  62. FRED-MD: A Monthly Database for Macroeconomic Research (2016)
  63. Inferential Theory for Factor Models of Large Dimensions (2003)
  64. Comparing predictive accuracy (1995)
  65. A Unified Framework for High-Dimensional Analysis of $ M $-Estimators with Decomposable Regularizers (2012)
  66. High-dimensional macroeconomic forecasting and variable selection via penalized regression (2019)
  67. Determining the number of factors from empirical distribution of eigenvalues (2010)
  68. S O F A R : large-scale association network learning (2019)
  69. I P A D : stable interpretable forecasting with knockoffs inference (2020)
  70. Arbitrage, Factor Structure and Mean-Variance Analysis in Large Asset Markets (1983)
  71. Performance Measurement With the Arbitrage Pricing Theory: A New Framework for Analysis (1986)
  72. Adaptive Robust Variable Selection (2014)
  73. Macro factors in bond risk premia (2009)
  74. Principal components estimation and identification of static factors (2013)
  75. Nonconcave penalized likelihood with N P -dimensionality (2011)
  76. Confidence intervals for low dimensional parameters in high dimensional linear models (2014)
  77. Nearly unbiased variable selection under minimax concave penalty (2014)
  78. Oracle efficient variable selection in random and fixed effects panel data models (2013)
  79. Consistent and conservative model selection with the adaptive lasso in stationary and nonstationary autoregressions (2016)
  80. Oracle inequalities for high dimensional vector autoregressions (2015)
  81. Confidence intervals and hypothesis testing for high-dimensional regression (2014)
  82. Variable selection via nonconcave penalized likelihood and its oracle properties (2001)
  83. Exponent of Cross-Sectional Dependence: Estimation and Inference (2016)
  84. Performance Measurement with the Arbitrage Pricing Theory: A New Framework for Analysis (1986)
  85. A Test for the Number of Factors in an Approximate Factor ModelA Test for the Number of Factors in an Approximate Factor Model (1993)
  86. Confidence intervals for diffusion index forecasts and inference with factor-augmented regressions (2006)
  87. ON ASYMPTOTICALLY OPTIMAL CONFIDENCE REGIONS AND TESTS FOR HIGH-DIMENSIONAL MODELS (2014)
  88. Robust high-dimensional factor models with applications to statistical machine learning (2018) — link
  89. A Testing Procedure for Determining the Number of Factors in Approximate Factor Models With Large Datasets (2010)
  90. On Consistency and Sparsity for Principal Components Analysis in High Dimensions (2009)
  91. Robust Principal Component Analysis? (2011)
  92. Principal Components and Regularized Estimation of Factor Models (2017) — link
  93. Estimating Latent Asset-Pricing Factors (2018)
  94. High dimensional covariance matrix estimation using a factor model (2008)
  95. A five-factor asset pricing model (2015)
  96. High-dimensional covariance matrix estimation in approximate factor models (2011)
  97. Hanson-Wright inequality and sub-gaussian concentration (2013)
  98. Asymptotics of the principal components estimator of large factor models with weakly influential factors (2012)
  99. Large covariance estimation by thresholding principal orthogonal complements (2013)
  100. Confidence intervals for low dimensional parameters in high dimensional linear models (2014)
  101. Determining the Number of Factors in the General Dynamic Factor Model (2007)
  102. Consistent Estimation of the Number of Dynamic Factors in a Large $N$ and $T$ Panel (2007)
  103. Selecting the Correct Number of Factors in Approximate Factor Models: The Large Panel Case With Group Bridge Estimators (2014)
  104. A Testing Procedure for Determining the Number of Factors in Approximate Factor Models With Large Datasets (2010)
  105. Spurious Factors in Linear Asset Pricing Models (2016)
  106. A Generalized Factor Model with Local Factors (2018)
  107. Robust Principal Component Analysis? (2011)
  108. Clustering Huge Number of Financial Time Series: A Panel Data Approach With High-Dimensional Predictors and Factor Structures (2017)
  109. A multilevel factor model: Identification, asymptotic theory and applications (2018)
  110. Panel Data Models With Interactive Fixed Effects (2009)
  111. Grouped graphical granger modeling for gene expression regulatory networks discovery (2009)
  112. Estimation and Inference in Large Heterogeneous Panels with a Multifactor Error Structure (2006)
  113. Inference in Group Factor Models with an Application to Mixed Frequency Data (2019)
  114. Estimation of latent factors for high-dimensional time series (2011)
  115. Estimation and inference of FAVAR models (2016)
  116. Estimation of weak factor models (2020)
  117. False discovery rate control via debiased lasso (2019)
  118. Gaussian graphical model estimation with false discovery rate control (2013)
  119. A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix (1987)
  120. Valid post-selection and postregularization inference: A n elementary, general approach (2015)
  121. Shrinking the cross-section (2020)
  122. A Bernstein type inequality and moderate deviations for weakly dependent sequences (2011)
  123. High-Dimensional Probability: An Introduction with Applications in Data Science (2018)
  124. Sub-Weibull distributions: generalizing sub-Gaussian and sub-Exponential properties to heavier-tailed distributions (2019) — link
  125. Theory and methods of panel data models with interactive effects (2014) — link
  126. Strong approximations of Martingale vectors and their applications in Markov-Chain adaptive designs (2004)
  127. An Introduction to Probability Theory and Its Applications Vol. 1 (3rd ed.) (1968)
  128. Mill’s Ratio for Multivariate Normal Distributions (1962)
  129. Investigating causal relations by econometric models and cross-spectral methods (1969)
  130. A Constrained $\ell_1$ Minimization Approach to Sparse Precision Matrix Estimation (2011)
  131. Strong approximations for nonconventional sums and almost sure limit theorems (2013)
  132. Estimation of autocovariance matrices for infinite dimensional vector linear process (2014)
  133. A Nonsymmetric Correlation Inequality for Gaussian Measure (1999)
  134. Inference in weak factor models (2021)
  135. Estimation of Sparsity-Induced Weak Factor Models (2023)
  136. New Introduction to Multiple Time Series Analysis (2005)
  137. Testing for high-dimensional network parameters in auto-regressive models (2019)
  138. Regularized Estimation and Testing for High-Dimensional Multi-Block Vector-Autoregressive Models (2017)
  139. N E T S : N etwork estimation for time series (2019)
  140. Network Granger Causality with Inherent Grouping Structure (2015)
  141. A Direct Estimation of High Dimensional Stationary Vector Autoregressions (2015)
  142. Estimation of large covariance and precision matrices from temporally dependent observations (2019)
  143. Bootstrapping autoregressions with conditional heteroskedasticity of unknown form (2004)
  144. Inference in VARs with conditional heteroskedasticity of unknown form (2016)
  145. Finite-Sample Properties of Percentile and Percentile-t Bootstrap Confidence Intervals for Impulse Responses (1999)
  146. Testing for linear vector autoregressive dynamics under multivariate generalized autoregressive heteroskedasticity (2009)
  147. Testing High Dimensional Forecasting via Interpretable Vector Autoregression (2020)
  148. A General Three-Series Theorem (1971)
  149. On convergence properties of sums of dependent random variables under second moment and covariance restrictions (2008)
  150. Bootstrap and wild bootstrap for high dimensional linear models (1993)
  151. The wild bootstrap, tamed at last (2008)
  152. Self-normalized Cram'er-type large deviations for independent random variables (2003)
  153. Testing for unit roots in heterogeneous panels (2003)
  154. Money, Income, and Causality (1972)
  155. On the G ranger condition for non-causality (1977)
  156. Granger causality and path diagrams for multivariate time series (2007)
  157. The Dantzig selector: statistical estimation when $p$ is much larger than $n$ (2007)
  158. Detecting groups in large vector autoregressions (2021)
  159. Granger Causality: A Review and Recent Advances (2021) — link
  160. Graphical modelling of multivariate time series (2012)
  161. Modeling gene expression regulatory networks with the sparse vector autoregressive model (2007)
  162. A unified theory of confidence regions and testing for highdimensional estimating equations (2018)
  163. Confidence intervals for parameters in high-dimensional sparse vector autoregression (2020) — link
  164. Sparse vector autoregressive modeling (2016)
  165. Low rank and structured modeling of high-dimensional vector autoregressions (2019)
  166. Regularized estimation in sparse high-dimensional time series models (2015)
  167. High-Dimensional Granger Causality Tests with an Application to VIX and News (2021)
  168. Skilled Mutual Fund Selection: False Discovery Control Under Dependence (2022)
  169. False Discoveries in Mutual Fund Performance:Measuring Luck in Estimated Alphas (2010)
  170. A Direct Approach to False Discovery Rates (2002)
  171. On Persistence in Mutual Fund Performance (1997)
  172. The Econometrics of Financial Markets (1997)
  173. Sparse and Stable Markowitz Portfolios (2009)
  174. Approaching Mean-Variance Efficiency for Large Portfolios (2019)
  175. Optimal Portfolio Choice with Parameter Uncertainty (2007)
  176. Panning for Gold: ‘Model‐X’ Knockoffs for High Dimensional Controlled Variable Selection (2018)
  177. Controlling the False Discovery Rate via Knockoffs (2015)
  178. A Nodewise Regression Approach to Estimating Large Portfolios (2021)
  179. IPAD : Stable Interpretable Forecasting with Knockoffs Inference (2020)
  180. High-dimensional graphs and variable selection with the lasso (2006)
  181. The Sampling Error in Estimates of Mean-Variance Efficient Portfolio Weights (1999)
  182. High-Dimensional Portfolio Selection with Cardinality Constraints (2023)
  183. Portfolio Construction with Hierarchical Momentum (2024)
  184. Factor Models for Portfolio Selection in Large Dimensions: The Good, the Better and the Ugly (2021)
  185. Portfolio Selection (1952)
  186. Taming the Factor Zoo: A Test of New Factors (2020)
  187. Making Distributionally Robust Portfolios Feasible in High Dimension (2025)
  188. Dissecting Characteristics Nonparametrically (2020)
  189. Dynamic Realized Minimum Variance Portfolio Models — → 2310.13511
  190. Optimal Shrinkage-Based Portfolio Selection in High Dimensions (2023)
  191. Inference on treatment effects after selection among high-dimensional controls (2014)
  192. Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps (2003)
  193. A Synthetic Regression Model for Large Portfolio Allocation (2022)
  194. Distributionally Robust Mean-Variance Portfolio Selection with Wasserstein Distances (2022)
  195. Vast Portfolio Selection With Gross-Exposure Constraints (2012)
  196. Nonlinear Shrinkage of the Covariance Matrix for Portfolio Selection: Markowitz Meets Goldilocks (2017)
  197. The arbitrage theory of capital asset pricing (1976)
  198. Arbitrage, factor structure, and mean-demand analysis on large asset markets (1983)
  199. Common risk factors in the returns on stocks and bonds (1993)
  200. How to use security analysis to improve portfolio selection (1973)
  201. On the inverse of the covariance matrix in portfolio analysis (1998)
  202. Thousands of Alpha Tests (2021)
  203. Sharp thresholds for high-dimensional and noisy sparsity recovery using L $_1$-constrained quadratic programming ( L asso) (2009)
  204. On model selection consistency of L asso (2006)
  205. Statistics for High-Dimensional Data: Methods, Theory and Applications (2011)
  206. Statistical Learning with Sparsity: The L asso and Generalizations (2015)
  207. A unified framework for high-dimensional analysis of $M$-estimators with decomposable regularizers (2012)
  208. Quadratic Shrinkage for Large Covariance Matrices (2022)
  209. Optimal Portfolio Using Factor Graphical Lasso (2024)
  210. Sparse Inverse Covariance Estimation with the Graphical Lasso (2007)
  211. Extended Bayesian Information Criteria for Gaussian Graphical Models (2010) — link
  212. Optimal Versus Naive Diversification:How Inefficient is the $1/N$ Portfolio Strategy? (2009)
  213. A Taxonomy of Anomalies and Their Trading Costs (2016)
  214. Trading Costs of Asset Pricing Anomalies (2014)
  215. Replicating Anomalies (2020)

1 of 223 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.