Transformer Models

On Evaluating Loss Functions for Stock Ranking: An Empirical Analysis With Transformer Model ArXiv ID: 2510.14156 “View on arXiv” Authors: Jan Kwiatkowski, Jarosław A. Chudziak Abstract Quantitative trading strategies rely on accurately ranking stocks to identify profitable investments. Effective portfolio management requires models that can reliably order future stock returns. Transformer models are promising for understanding financial time series, but how different training loss functions affect their ability to rank stocks well is not yet fully understood. Financial markets are challenging due to their changing nature and complex relationships between stocks. Standard loss functions, which aim for simple prediction accuracy, often aren’t enough. They don’t directly teach models to learn the correct order of stock returns. While many advanced ranking losses exist from fields such as information retrieval, there hasn’t been a thorough comparison to see how well they work for ranking financial returns, especially when used with modern Transformer models for stock selection. This paper addresses this gap by systematically evaluating a diverse set of advanced loss functions including pointwise, pairwise, listwise for daily stock return forecasting to facilitate rank-based portfolio selection on S&P 500 data. We focus on assessing how each loss function influences the model’s ability to discern profitable relative orderings among assets. Our research contributes a comprehensive benchmark revealing how different loss functions impact a model’s ability to learn cross-sectional and temporal patterns crucial for portfolio selection, thereby offering practical guidance for optimizing ranking-based trading strategies. ...

Quantifying Semantic Shift in Financial NLP: Robust Metrics for Market Prediction Stability ArXiv ID: 2510.00205 “View on arXiv” Authors: Zhongtian Sun, Chenghao Xiao, Anoushka Harit, Jongmin Yu Abstract Financial news is essential for accurate market prediction, but evolving narratives across macroeconomic regimes introduce semantic and causal drift that weaken model reliability. We present an evaluation framework to quantify robustness in financial NLP under regime shifts. The framework defines four metrics: (1) Financial Causal Attribution Score (FCAS) for alignment with causal cues, (2) Patent Cliff Sensitivity (PCS) for sensitivity to semantic perturbations, (3) Temporal Semantic Volatility (TSV) for drift in latent text representations, and (4) NLI-based Logical Consistency Score (NLICS) for entailment coherence. Applied to LSTM and Transformer models across four economic periods (pre-COVID, COVID, post-COVID, and rate hike), the metrics reveal performance degradation during crises. Semantic volatility and Jensen-Shannon divergence correlate with prediction error. Transformers are more affected by drift, while feature-enhanced variants improve generalisation. A GPT-4 case study confirms that alignment-aware models better preserve causal and logical consistency. The framework supports auditability, stress testing, and adaptive retraining in financial AI systems. ...

Language of Persuasion and Misrepresentation in Business Communication: A Textual Detection Approach ArXiv ID: 2508.09935 “View on arXiv” Authors: Sayem Hossen, Monalisa Moon Joti, Md. Golam Rashed Abstract Business communication digitisation has reorganised the process of persuasive discourse, which allows not only greater transparency but also advanced deception. This inquiry synthesises classical rhetoric and communication psychology with linguistic theory and empirical studies in the financial reporting, sustainability discourse, and digital marketing to explain how deceptive language can be systematically detected using persuasive lexicon. In controlled settings, detection accuracies of greater than 99% were achieved by using computational textual analysis as well as personalised transformer models. However, reproducing this performance in multilingual settings is also problematic and, to a large extent, this is because it is not easy to find sufficient data, and because few multilingual text-processing infrastructures are in place. This evidence shows that there has been an increasing gap between the theoretical representations of communication and those empirically approximated, and therefore, there is a need to have strong automatic text-identification systems where AI-based discourse is becoming more realistic in communicating with humans. ...

FinAI-BERT: A Transformer-Based Model for Sentence-Level Detection of AI Disclosures in Financial Reports ArXiv ID: 2507.01991 “View on arXiv” Authors: Muhammad Bilal Zafar Abstract The proliferation of artificial intelligence (AI) in financial services has prompted growing demand for tools that can systematically detect AI-related disclosures in corporate filings. While prior approaches often rely on keyword expansion or document-level classification, they fall short in granularity, interpretability, and robustness. This study introduces FinAI-BERT, a domain-adapted transformer-based language model designed to classify AI-related content at the sentence level within financial texts. The model was fine-tuned on a manually curated and balanced dataset of 1,586 sentences drawn from 669 annual reports of U.S. banks (2015 to 2023). FinAI-BERT achieved near-perfect classification performance (accuracy of 99.37 percent, F1 score of 0.993), outperforming traditional baselines such as Logistic Regression, Naive Bayes, Random Forest, and XGBoost. Interpretability was ensured through SHAP-based token attribution, while bias analysis and robustness checks confirmed the model’s stability across sentence lengths, adversarial inputs, and temporal samples. Theoretically, the study advances financial NLP by operationalizing fine-grained, theme-specific classification using transformer architectures. Practically, it offers a scalable, transparent solution for analysts, regulators, and scholars seeking to monitor the diffusion and framing of AI across financial institutions. ...

Transformers Beyond Order: A Chaos-Markov-Gaussian Framework for Short-Term Sentiment Forecasting of Any Financial OHLC timeseries Data ArXiv ID: 2506.17244 “View on arXiv” Authors: Arif Pathan Abstract Short-term sentiment forecasting in financial markets (e.g., stocks, indices) is challenging due to volatility, non-linearity, and noise in OHLC (Open, High, Low, Close) data. This paper introduces a novel CMG (Chaos-Markov-Gaussian) framework that integrates chaos theory, Markov property, and Gaussian processes to improve prediction accuracy. Chaos theory captures nonlinear dynamics; the Markov chain models regime shifts; Gaussian processes add probabilistic reasoning. We enhance the framework with transformer-based deep learning models to capture temporal patterns efficiently. The CMG Framework is designed for fast, resource-efficient, and accurate forecasting of any financial instrument’s OHLC time series. Unlike traditional models that require heavy infrastructure and instrument-specific tuning, CMG reduces overhead and generalizes well. We evaluate the framework on market indices, forecasting sentiment for the next trading day’s first quarter. A comparative study against statistical, ML, and DL baselines trained on the same dataset with no feature engineering shows CMG consistently outperforms in accuracy and efficiency, making it valuable for analysts and financial institutions. ...

Explainable-AI powered stock price prediction using time series transformers: A Case Study on BIST100 ArXiv ID: 2506.06345 “View on arXiv” Authors: Sukru Selim Calik, Andac Akyuz, Zeynep Hilal Kilimci, Kerem Colak Abstract Financial literacy is increasingly dependent on the ability to interpret complex financial data and utilize advanced forecasting tools. In this context, this study proposes a novel approach that combines transformer-based time series models with explainable artificial intelligence (XAI) to enhance the interpretability and accuracy of stock price predictions. The analysis focuses on the daily stock prices of the five highest-volume banks listed in the BIST100 index, along with XBANK and XU100 indices, covering the period from January 2015 to March 2025. Models including DLinear, LTSNet, Vanilla Transformer, and Time Series Transformer are employed, with input features enriched by technical indicators. SHAP and LIME techniques are used to provide transparency into the influence of individual features on model outputs. The results demonstrate the strong predictive capabilities of transformer models and highlight the potential of interpretable machine learning to empower individuals in making informed investment decisions and actively engaging in financial markets. ...

Trading Under Uncertainty: A Distribution-Based Strategy for Futures Markets Using FutureQuant Transformer ArXiv ID: 2505.05595 “View on arXiv” Authors: Wenhao Guo, Yuda Wang, Zeqiao Huang, Changjiang Zhang, Shumin ma Abstract In the complex landscape of traditional futures trading, where vast data and variables like real-time Limit Order Books (LOB) complicate price predictions, we introduce the FutureQuant Transformer model, leveraging attention mechanisms to navigate these challenges. Unlike conventional models focused on point predictions, the FutureQuant model excels in forecasting the range and volatility of future prices, thus offering richer insights for trading strategies. Its ability to parse and learn from intricate market patterns allows for enhanced decision-making, significantly improving risk management and achieving a notable average gain of 0.1193% per 30-minute trade over state-of-the-art models with a simple algorithm using factors such as RSI, ATR, and Bollinger Bands. This innovation marks a substantial leap forward in predictive analytics within the volatile domain of futures trading. ...

Hedging with Sparse Reward Reinforcement Learning ArXiv ID: 2503.04218 “View on arXiv” Authors: Unknown Abstract Derivatives, as a critical class of financial instruments, isolate and trade the price attributes of risk assets such as stocks, commodities, and indices, aiding risk management and enhancing market efficiency. However, traditional hedging models, constrained by assumptions such as continuous trading and zero transaction costs, fail to satisfy risk control requirements in complex and uncertain real-world markets. With advances in computing technology and deep learning, data-driven trading strategies are becoming increasingly prevalent. This thesis proposes a derivatives hedging framework integrating deep learning and reinforcement learning. The framework comprises a probabilistic forecasting model and a hedging agent, enabling market probability prediction, derivative pricing, and hedging. Specifically, we design a spatiotemporal attention-based probabilistic financial time series forecasting Transformer to address the scarcity of derivatives hedging data. A low-rank attention mechanism compresses high-dimensional assets into a low-dimensional latent space, capturing nonlinear asset relationships. The Transformer models sequential dependencies within this latent space, improving market probability forecasts and constructing an online training environment for downstream hedging tasks. Additionally, we incorporate generalized geometric Brownian motion to develop a risk-neutral pricing approach for derivatives. We model derivatives hedging as a reinforcement learning problem with sparse rewards and propose a behavior cloning-based recurrent proximal policy optimization (BC-RPPO) algorithm. This pretraining-finetuning framework significantly enhances the hedging agent’s performance. Numerical experiments in the U.S. and Chinese financial markets demonstrate our method’s superiority over traditional approaches. ...

Hidformer: Transformer-Style Neural Network in Stock Price Forecasting ArXiv ID: 2412.19932 “View on arXiv” Authors: Unknown Abstract This paper investigates the application of Transformer-based neural networks to stock price forecasting, with a special focus on the intersection of machine learning techniques and financial market analysis. The evolution of Transformer models, from their inception to their adaptation for time series analysis in financial contexts, is reviewed and discussed. Central to our study is the exploration of the Hidformer model, which is currently recognized for its promising performance in time series prediction. The primary aim of this paper is to determine whether Hidformer will also prove itself in the task of stock price prediction. This slightly modified model serves as the framework for our experiments, integrating the principles of technical analysis with advanced machine learning concepts to enhance stock price prediction accuracy. We conduct an evaluation of the Hidformer model’s performance, using a set of criteria to determine its efficacy. Our findings offer additional insights into the practical application of Transformer architectures in financial time series forecasting, highlighting their potential to improve algorithmic trading strategies, including human decision making. ...

Dynamic ETF Portfolio Optimization Using enhanced Transformer-Based Models for Covariance and Semi-Covariance Prediction(Work in Progress) ArXiv ID: 2411.19649 “View on arXiv” Authors: Unknown Abstract This study explores the use of Transformer-based models to predict both covariance and semi-covariance matrices for ETF portfolio optimization. Traditional portfolio optimization techniques often rely on static covariance estimates or impose strict model assumptions, which may fail to capture the dynamic and non-linear nature of market fluctuations. Our approach leverages the power of Transformer models to generate adaptive, real-time predictions of asset covariances, with a focus on the semi-covariance matrix to account for downside risk. The semi-covariance matrix emphasizes negative correlations between assets, offering a more nuanced approach to risk management compared to traditional methods that treat all volatility equally. Through a series of experiments, we demonstrate that Transformer-based predictions of both covariance and semi-covariance significantly enhance portfolio performance. Our results show that portfolios optimized using the semi-covariance matrix outperform those optimized with the standard covariance matrix, particularly in volatile market conditions. Moreover, the use of the Sortino ratio, a risk-adjusted performance metric that focuses on downside risk, further validates the effectiveness of our approach in managing risk while maximizing returns. These findings have important implications for asset managers and investors, offering a dynamic, data-driven framework for portfolio construction that adapts more effectively to shifting market conditions. By integrating Transformer-based models with the semi-covariance matrix for improved risk management, this research contributes to the growing field of machine learning in finance and provides valuable insights for optimizing ETF portfolios. ...