Paper: arXiv 2412.10540
Authors: Soroush Omranpour, Guillaume Rabusseau, Reihaneh Rabbany
Abstract
In this paper, we tackle the challenge of predicting stock movements in financial markets by introducing Higher Order Transformers, a novel architecture designed for processing multivariate time-series data. We extend the self-attention mechanism and the transformer architecture to a higher order, effectively capturing complex market dynamics across time and variables. To manage computational complexity, we propose a low-rank approximation of the potentially large attention tensor using tensor decomposition and employ kernel attention, reducing complexity to linear with respect to the data size. Additionally, we present an encoder-decoder model that integrates technical and fundamental analysis, utilizing multimodal signals from historical prices and related tweets. Our experiments on the Stocknet dataset demonstrate the effectiveness of our method, highlighting its potential for enhancing stock movement prediction in financial markets.
Complexity vs Empirical Score
- Math Complexity: 8.5/10
- Empirical Rigor: 5.5/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper introduces a novel, mathematically complex higher-order transformer architecture using tensor decomposition and kernel attention, with rigorous theoretical formulations. It provides empirical validation on a specific dataset (Stocknet) with multimodal data (prices and tweets), though the summary lacks details on code, backtesting specifics, or extensive statistical metrics, making it more research-oriented than immediately backtest-ready.
Research Flowchart
flowchart TD A["Research Goal: Enhance Stock Movement Prediction<br>on Multimodal Time-Series Data"] --> B["Data/Inputs"] B --> B1["Historical Prices<br>Technical Analysis"] B --> B2["Related Tweets<br>Fundamental Analysis"] B1 & B2 --> C["Methodology: Higher Order Transformers"] C --> D["Computational Processes"] D --> D1["Extend Self-Attention to Higher Order"] D --> D2["Low-Rank Tensor Decomposition<br>+ Kernel Attention<br>Complexity: Linear O(N)"] D1 & D2 --> E["Key Findings/Outcomes"] E --> E1["Effective Capture of Complex<br>Market Dynamics"] E --> E2["Enhanced Prediction Accuracy<br>on Stocknet Dataset"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes Higher‑Order Transformers, a novel architecture for processing higher‑order multivariate time‑series data, together with a multimodal encoder‑decoder that integrates price and tweet information.
Source sentence
We introduce Higher-Order Transformers, a novel architecture tailored for processing higher-order data (with more than two dimensions where the last dimension works as the feature dimension).
Assumes 2
- Stock movement labels are defined by a threshold: movements ≥0.55% are positive and ≤0.5% are negative. data
Source sentence
Samples are labeled based on the movement percentage of the closing price, with movements ≥0.55% labeled as positive and ≤0.5% as negative.
- Samples lacking either price data or tweets on a trading day are discarded to ensure data consistency. data
Source sentence
Samples lacking either prices or tweets on trading days are discarded to ensure data consistency.
Data
- Stocknet market-prices public
Source sentence
We demonstrate the capability of Higher Order Transformers on stock market movement classification using a multimodal dataset called Stocknet
Twitter / X Yahoo Finance
Costs
Not applicable
Finds 1
- The proposed model outperforms all baselines except NL‑LSTM, which achieves higher accuracy, F1 and MCC. performance
Source sentence
The results, as summarized in Table 1, demonstrate the superior performance of our model over all of the existing baselines except NL-LSTM, which significantly outperforms the rest of the methods in terms of accuracy, F1, and MCC.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 43 related papers). Coverage is partial, so this is neither replication nor refutation.
2 papers in the archive : From News to Returns: A Granger-Causal Hypergraph …
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (39)
- Predicting the oil prices: do technical indicators help? (2016)
- Forecasting directional movement of Forex data using LSTM with technical and macroeconomic indicators (2021)
- Generalized autoregressive conditional heteroskedasticity (1986)
- Stock trend prediction using news sentiment analysis (2016) — link
- Fundamental analysis, future earnings, and stock prices (1997)
- Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction (2018)
- Deep learning for event-driven stock prediction (2015)
- Stock movement prediction from tweets and historical prices (2018)
and 31 more
- Temporal relational ranking for stock prediction (2019)
- Spatiotemporal hypergraph convolution network for stock movement forecasting (2020)
- Hats: A hierarchical graph attention network for stock movement prediction (2019) — link
- Exploring graph neural networks for stock market predictions with rolling window analysis (2019) — link
- FinKG: A Core Financial Knowledge Graph for Financial Analysis (2023)
- Stock movement prediction and portfolio management via multimodal learning with transformer (2021)
- A multimodal event-driven LSTM model for stock prediction using online news (2020)
- Deep attentive learning for stock movement prediction from social media text and company correlations (2020)
- Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs (2022) — link
- Rethinking Attention with Performers (2022) — link
- RoFormer: Enhanced Transformer with Rotary Position Embedding (2023) — link
- Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Prediction (2019) — link
- Fundamental Analysis based Neural Network for Stock Movement Prediction (2022) — link
- LLaMA: Open and Efficient Foundation Language Models (2023) — link
- Root Mean Square Layer Normalization (2019) — link
- The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges — → 2304.05351
- FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
- Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting (2024) — link
- Vision Transformers Need Registers (2024) — link
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2021) — link
- Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2021) — link
- Attention Is All You Need (2023) — link
- Smoothing, forecasting and prediction of discrete time series (2004)
- Astock: A New Dataset and Automated Stock Trading based on Stock-specific News Analyzing Model (2022) — link
- Transformer-Based Deep Learning Model for Stock Price Prediction: A Case Study on Bangladesh Stock Market (2023) — link
- Transformer-based attention network for stock movement prediction (2022) — link
- Transformer-Based Capsule Network For Stock Movement Prediction (2019) — link
- UniT: Multimodal Multitask Learning with a Unified Transformer (2021) — link
- Multimodal Transformer for Unaligned Multimodal Language Sequences (2019) — link
- Accurate Stock Movement Prediction with Self-supervised Learning from Sparse Noisy Tweets (2022) — link
1 of 39 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.