Machine Learning

A Dynamic Model of Private Asset Allocation ArXiv ID: 2503.01099 “View on arXiv” Authors: Unknown Abstract We build a state-of-the-art dynamic model of private asset allocation that considers five key features of private asset markets: (1) the illiquid nature of private assets, (2) timing lags between capital commitments, capital calls, and eventual distributions, (3) time-varying business cycle conditions, (4) serial correlation in observed private asset returns, and (5) regulatory constraints on certain institutional investors’ portfolio choices. We use cutting-edge machine learning methods to quantify the optimal investment policies over the life cycle of a fund. Moreover, our model offers regulators a tool for precisely quantifying the trade-offs when setting risk-based capital charges. ...

Time Series Feature Redundancy Paradox: An Empirical Study Based on Mortgage Default Prediction ArXiv ID: 2501.00034 “View on arXiv” Authors: Unknown Abstract With the widespread application of machine learning in financial risk management, conventional wisdom suggests that longer training periods and more feature variables contribute to improved model performance. This paper, focusing on mortgage default prediction, empirically discovers a phenomenon that contradicts traditional knowledge: in time series prediction, increased training data timespan and additional non-critical features actually lead to significant deterioration in prediction effectiveness. Using Fannie Mae’s mortgage data, the study compares predictive performance across different time window lengths (2012-2022) and feature combinations, revealing that shorter time windows (such as single-year periods) paired with carefully selected key features yield superior prediction results. The experimental results indicate that extended time spans may introduce noise from historical data and outdated market patterns, while excessive non-critical features interfere with the model’s learning of core default factors. This research not only challenges the traditional “more is better” approach in data modeling but also provides new insights and practical guidance for feature selection and time window optimization in financial risk prediction. ...

A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation ArXiv ID: 2412.15298 “View on arXiv” Authors: Unknown Abstract We argue that the Declarative Self-improving Python (DSPy) optimizers are a way to align the large language model (LLM) prompts and their evaluations to the human annotations. We present a comparative analysis of five teleprompter algorithms, namely, Cooperative Prompt Optimization (COPRO), Multi-Stage Instruction Prompt Optimization (MIPRO), BootstrapFewShot, BootstrapFewShot with Optuna, and K-Nearest Neighbor Few Shot, within the DSPy framework with respect to their ability to align with human evaluations. As a concrete example, we focus on optimizing the prompt to align hallucination detection (using LLM as a judge) to human annotated ground truth labels for a publicly available benchmark dataset. Our experiments demonstrate that optimized prompts can outperform various benchmark methods to detect hallucination, and certain telemprompters outperform the others in at least these experiments. ...

AI-Enhanced Factor Analysis for Predicting S&P 500 Stock Dynamics ArXiv ID: 2412.12438 “View on arXiv” Authors: Unknown Abstract This project investigates the interplay of technical, market, and statistical factors in predicting stock market performance, with a primary focus on S&P 500 companies. Utilizing a comprehensive dataset spanning multiple years, the analysis constructs advanced financial metrics, such as momentum indicators, volatility measures, and liquidity adjustments. The machine learning framework is employed to identify patterns, relationships, and predictive capabilities of these factors. The integration of traditional financial analytics with machine learning enables enhanced predictive accuracy, offering valuable insights into market behavior and guiding investment strategies. This research highlights the potential of combining domain-specific financial expertise with modern computational tools to address complex market dynamics. ...

S&P 500 Trend Prediction ArXiv ID: 2412.11462 “View on arXiv” Authors: Unknown Abstract This project aims to predict short-term and long-term upward trends in the S&P 500 index using machine learning models and feature engineering based on the “101 Formulaic Alphas” methodology. The study employed multiple models, including Logistic Regression, Decision Trees, Random Forests, Neural Networks, K-Nearest Neighbors (KNN), and XGBoost, to identify market trends from historical stock data collected from Yahoo! Finance. Data preprocessing involved handling missing values, standardization, and iterative feature selection to ensure relevance and variability. For short-term predictions, KNN emerged as the most effective model, delivering robust performance with high recall for upward trends, while for long-term forecasts, XGBoost demonstrated the highest accuracy and AUC scores after hyperparameter tuning and class imbalance adjustments using SMOTE. Feature importance analysis highlighted the dominance of momentum-based and volume-related indicators in driving predictions. However, models exhibited limitations such as overfitting and low recall for positive market movements, particularly in imbalanced datasets. The study concludes that KNN is ideal for short-term alerts, whereas XGBoost is better suited for long-term trend forecasting. Future enhancements could include advanced architectures like Long Short-Term Memory (LSTM) networks and further feature refinement to improve precision and generalizability. These findings contribute to developing reliable machine learning tools for market trend prediction and investment decision-making. ...

PolyModel for Hedge Funds’ Portfolio Construction Using Machine Learning ArXiv ID: 2412.11019 “View on arXiv” Authors: Unknown Abstract The domain of hedge fund investments is undergoing significant transformation, influenced by the rapid expansion of data availability and the advancement of analytical technologies. This study explores the enhancement of hedge fund investment performance through the integration of machine learning techniques, the application of PolyModel feature selection, and the analysis of fund size. We address three critical questions: (1) the effect of machine learning on trading performance, (2) the role of PolyModel feature selection in fund selection and performance, and (3) the comparative reliability of larger versus smaller funds. Our findings offer compelling insights. We observe that while machine learning techniques enhance cumulative returns, they also increase annual volatility, indicating variability in performance. PolyModel feature selection proves to be a robust strategy, with approaches that utilize a comprehensive set of features for fund selection outperforming more selective methodologies. Notably, Long-Term Stability (LTS) effectively manages portfolio volatility while delivering favorable returns. Contrary to popular belief, our results suggest that larger funds do not consistently yield better investment outcomes, challenging the assumption of their inherent reliability. This research highlights the transformative impact of data-driven approaches in the hedge fund investment arena and provides valuable implications for investors and asset managers. By leveraging machine learning and PolyModel feature selection, investors can enhance portfolio optimization and reassess the dependability of larger funds, leading to more informed investment strategies. ...

A Consolidated Volatility Prediction with Back Propagation Neural Network and Genetic Algorithm ArXiv ID: 2412.07223 “View on arXiv” Authors: Unknown Abstract This paper provides a unique approach with AI algorithms to predict emerging stock markets volatility. Traditionally, stock volatility is derived from historical volatility,Monte Carlo simulation and implied volatility as well. In this paper, the writer designs a consolidated model with back-propagation neural network and genetic algorithm to predict future volatility of emerging stock markets and found that the results are quite accurate with low errors. ...

FinML-Chain: A Blockchain-Integrated Dataset for Enhanced Financial Machine Learning ArXiv ID: 2411.16277 “View on arXiv” Authors: Unknown Abstract Machine learning is critical for innovation and efficiency in financial markets, offering predictive models and data-driven decision-making. However, challenges such as missing data, lack of transparency, untimely updates, insecurity, and incompatible data sources limit its effectiveness. Blockchain technology, with its transparency, immutability, and real-time updates, addresses these challenges. We present a framework for integrating high-frequency on-chain data with low-frequency off-chain data, providing a benchmark for addressing novel research questions in economic mechanism design. This framework generates modular, extensible datasets for analyzing economic mechanisms such as the Transaction Fee Mechanism, enabling multi-modal insights and fairness-driven evaluations. Using four machine learning techniques, including linear regression, deep neural networks, XGBoost, and LSTM models, we demonstrate the framework’s ability to produce datasets that advance financial research and improve understanding of blockchain-driven systems. Our contributions include: (1) proposing a research scenario for the Transaction Fee Mechanism and demonstrating how the framework addresses previously unexplored questions in economic mechanism design; (2) providing a benchmark for financial machine learning by open-sourcing a sample dataset generated by the framework and the code for the pipeline, enabling continuous dataset expansion; and (3) promoting reproducibility, transparency, and collaboration by fully open-sourcing the framework and its outputs. This initiative supports researchers in extending our work and developing innovative financial machine-learning models, fostering advancements at the intersection of machine learning, blockchain, and economics. ...

A Deep Learning Approach to Predict the Fall [“of Price”] of Cryptocurrency Long Before its Actual Fall ArXiv ID: 2411.13615 “View on arXiv” Authors: Unknown Abstract In modern times, the cryptocurrency market is one of the world’s most rapidly rising financial markets. The cryptocurrency market is regarded to be more volatile and illiquid than traditional markets such as equities, foreign exchange, and commodities. The risk of this market creates an uncertain condition among the investors. The purpose of this research is to predict the magnitude of the risk factor of the cryptocurrency market. Risk factor is also called volatility. Our approach will assist people who invest in the cryptocurrency market by overcoming the problems and difficulties they experience. Our approach starts with calculating the risk factor of the cryptocurrency market from the existing parameters. In twenty elements of the cryptocurrency market, the risk factor has been predicted using different machine learning algorithms such as CNN, LSTM, BiLSTM, and GRU. All of the models have been applied to the calculated risk factor parameter. A new model has been developed to predict better than the existing models. Our proposed model gives the highest RMSE value of 1.3229 and the lowest RMSE value of 0.0089. Following our model, it will be easier for investors to trade in complicated and challenging financial assets like bitcoin, Ethereum, dogecoin, etc. Where the other existing models, the highest RMSE was 14.5092, and the lower was 0.02769. So, the proposed model performs much better than models with proper generalization. Using our approach, it will be easier for investors to trade in complicated and challenging financial assets like Bitcoin, Ethereum, and Dogecoin. ...

European Option Pricing in Regime Switching Framework via Physics-Informed Residual Learning ArXiv ID: 2410.10474 “View on arXiv” Authors: Unknown Abstract In this article, we employ physics-informed residual learning (PIRL) and propose a pricing method for European options under a regime-switching framework, where closed-form solutions are not available. We demonstrate that the proposed approach serves an efficient alternative to competing pricing techniques for regime-switching models in the literature. Specifically, we demonstrate that PIRLs eliminate the need for retraining and become nearly instantaneous once trained, thus, offering an efficient and flexible tool for pricing options across a broad range of specifications and parameters. ...