Paper: arXiv 2412.10540

Authors: Soroush Omranpour, Guillaume Rabusseau, Reihaneh Rabbany

Abstract

In this paper, we tackle the challenge of predicting stock movements in financial markets by introducing Higher Order Transformers, a novel architecture designed for processing multivariate time-series data. We extend the self-attention mechanism and the transformer architecture to a higher order, effectively capturing complex market dynamics across time and variables. To manage computational complexity, we propose a low-rank approximation of the potentially large attention tensor using tensor decomposition and employ kernel attention, reducing complexity to linear with respect to the data size. Additionally, we present an encoder-decoder model that integrates technical and fundamental analysis, utilizing multimodal signals from historical prices and related tweets. Our experiments on the Stocknet dataset demonstrate the effectiveness of our method, highlighting its potential for enhancing stock movement prediction in financial markets.

Complexity vs Empirical Score

  • Math Complexity: 8.5/10
  • Empirical Rigor: 5.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper introduces a novel, mathematically complex higher-order transformer architecture using tensor decomposition and kernel attention, with rigorous theoretical formulations. It provides empirical validation on a specific dataset (Stocknet) with multimodal data (prices and tweets), though the summary lacks details on code, backtesting specifics, or extensive statistical metrics, making it more research-oriented than immediately backtest-ready.

Research Flowchart

  flowchart TD
  A["Research Goal: Enhance Stock Movement Prediction<br>on Multimodal Time-Series Data"] --> B["Data/Inputs"]
  B --> B1["Historical Prices<br>Technical Analysis"]
  B --> B2["Related Tweets<br>Fundamental Analysis"]
  B1 & B2 --> C["Methodology: Higher Order Transformers"]
  C --> D["Computational Processes"]
  D --> D1["Extend Self-Attention to Higher Order"]
  D --> D2["Low-Rank Tensor Decomposition<br>+ Kernel Attention<br>Complexity: Linear O(N)"]
  D1 & D2 --> E["Key Findings/Outcomes"]
  E --> E1["Effective Capture of Complex<br>Market Dynamics"]
  E --> E2["Enhanced Prediction Accuracy<br>on Stocknet Dataset"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes Higher‑Order Transformers, a novel architecture for processing higher‑order multivariate time‑series data, together with a multimodal encoder‑decoder that integrates price and tweet information.

Source sentence
We introduce Higher-Order Transformers, a novel architecture tailored for processing higher-order data (with more than two dimensions where the last dimension works as the feature dimension).

Assumes 2

  • Stock movement labels are defined by a threshold: movements ≥0.55% are positive and ≤0.5% are negative. data
    Source sentence
    Samples are labeled based on the movement percentage of the closing price, with movements ≥0.55% labeled as positive and ≤0.5% as negative.
  • Samples lacking either price data or tweets on a trading day are discarded to ensure data consistency. data
    Source sentence
    Samples lacking either prices or tweets on trading days are discarded to ensure data consistency.

Data

  • Stocknet · Yahoo Finance and Twitter (crawled) market-prices public
    two years (01/01/2014 to 01/01/2016) · 88 stocks
    Source sentence
    We demonstrate the capability of Higher Order Transformers on stock market movement classification using a multimodal dataset called Stocknet

Recognised sources: Twitter / X Yahoo Finance

Costs

Not applicable The paper reports no trading or execution result.

Finds 1

  • The proposed model outperforms all baselines except NL‑LSTM, which achieves higher accuracy, F1 and MCC. performance
    Source sentence
    The results, as summarized in Table 1, demonstrate the superior performance of our model over all of the existing baselines except NL-LSTM, which significantly outperforms the rest of the methods in terms of accuracy, F1, and MCC.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 43 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 2 papers in the archive (2 by independent authors): From News to Returns: A Granger-Causal Hypergraph … (used as a baseline)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (39)

  1. Predicting the oil prices: do technical indicators help? (2016)
  2. Forecasting directional movement of Forex data using LSTM with technical and macroeconomic indicators (2021)
  3. Generalized autoregressive conditional heteroskedasticity (1986)
  4. Stock trend prediction using news sentiment analysis (2016) — link
  5. Fundamental analysis, future earnings, and stock prices (1997)
  6. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction (2018)
  7. Deep learning for event-driven stock prediction (2015)
  8. Stock movement prediction from tweets and historical prices (2018)
and 31 more
  1. Temporal relational ranking for stock prediction (2019)
  2. Spatiotemporal hypergraph convolution network for stock movement forecasting (2020)
  3. Hats: A hierarchical graph attention network for stock movement prediction (2019) — link
  4. Exploring graph neural networks for stock market predictions with rolling window analysis (2019) — link
  5. FinKG: A Core Financial Knowledge Graph for Financial Analysis (2023)
  6. Stock movement prediction and portfolio management via multimodal learning with transformer (2021)
  7. A multimodal event-driven LSTM model for stock prediction using online news (2020)
  8. Deep attentive learning for stock movement prediction from social media text and company correlations (2020)
  9. Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs (2022) — link
  10. Rethinking Attention with Performers (2022) — link
  11. RoFormer: Enhanced Transformer with Rotary Position Embedding (2023) — link
  12. Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Prediction (2019) — link
  13. Fundamental Analysis based Neural Network for Stock Movement Prediction (2022) — link
  14. LLaMA: Open and Efficient Foundation Language Models (2023) — link
  15. Root Mean Square Layer Normalization (2019) — link
  16. The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges — → 2304.05351
  17. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
  18. Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting (2024) — link
  19. Vision Transformers Need Registers (2024) — link
  20. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  21. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2021) — link
  22. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2021) — link
  23. Attention Is All You Need (2023) — link
  24. Smoothing, forecasting and prediction of discrete time series (2004)
  25. Astock: A New Dataset and Automated Stock Trading based on Stock-specific News Analyzing Model (2022) — link
  26. Transformer-Based Deep Learning Model for Stock Price Prediction: A Case Study on Bangladesh Stock Market (2023) — link
  27. Transformer-based attention network for stock movement prediction (2022) — link
  28. Transformer-Based Capsule Network For Stock Movement Prediction (2019) — link
  29. UniT: Multimodal Multitask Learning with a Unified Transformer (2021) — link
  30. Multimodal Transformer for Unaligned Multimodal Language Sequences (2019) — link
  31. Accurate Stock Movement Prediction with Self-supervised Learning from Sparse Noisy Tweets (2022) — link

1 of 39 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.