Paper: arXiv 2509.19628

Authors: Ross Koval, Nicholas Andrews, Xifeng Yan

Abstract

Text and time series data offer complementary views of financial markets: news articles provide narrative context about company events, while stock prices reflect how markets react to those events. However, despite their complementary nature, effectively integrating these interleaved modalities for improved forecasting remains challenging. In this work, we propose a unified neural architecture that models these interleaved sequences using modality-specific experts, allowing the model to learn unique time series patterns, while still enabling joint reasoning across modalities and preserving pretrained language understanding capabilities. To further improve multimodal understanding, we introduce a cross-modal alignment framework with a salient token weighting mechanism that learns to align representations across modalities with a focus on the most informative tokens. We demonstrate the effectiveness of our approach on a large-scale financial forecasting task, achieving state-of-the-art performance across a wide variety of strong unimodal and multimodal baselines. We develop an interpretability method that reveals insights into the value of time series-context and reinforces the design of our cross-modal alignment objective. Finally, we demonstrate that these improvements translate to meaningful economic gains in investment simulations.

Complexity vs Empirical Score

  • Math Complexity: 6.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Holy Grail — high math complexity, high empirical rigor

Why this score: The paper employs advanced neural architectures like mixture-of-experts (MoE) and cross-modal alignment frameworks with specific objective functions (SALMON, STW), indicating significant math complexity. It also demonstrates empirical rigor through a large-scale forecasting task, a released codebase, and investment simulations with economic gains, making it backtest-ready.

Research Flowchart

  flowchart TD
  G["Research Goal<br>How to effectively integrate interleaved<br>text & time series for financial forecasting?"] --> D
  subgraph D ["Inputs & Preprocessing"]
      D1["Text Data<br>(News Articles)"] --> D3["Interleaved Sequences"]
      D2["Time Series Data<br>(Stock Prices)"] --> D3
  end
  D --> M
  subgraph M ["Methodology"]
      direction LR
      M1["Modality-Specific Experts"] --> M2["Cross-Modal Alignment<br>with Salient Token Weighting"] --> M3["Joint Reasoning &<br>Pretrained Language Modeling"]
  end
  M --> F["Computational Process<br>Multimodal LLM Architecture"]
  F --> O
  subgraph O ["Key Findings & Outcomes"]
      O1["State-of-the-Art<br>Forecasting Performance"]
      O2["Interpretability<br>Insights"]
      O3["Meaningful Economic Gains<br>in Investment Simulations"]
  end
  O --> R["Research Impact<br>Validates Modality-Specific Experts &<br>Cross-Modal Alignment for Finance"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a unified multimodal architecture with modality‑specific experts (MSE‑ITT) and a cross‑modal alignment framework with salient token weighting (SALMON).

Source sentence
We design a multimodal architecture (MSE‑ITT) that can effectively model interleaved sequences of text and time series, demonstrating state-of-the-art performance on a challenging financial forecasting task...

Assumes 5

  • The forecasting task is a binary classification of stock price direction over short‑term (7D) and long‑term (30D) horizons. data
    Source sentence
    we adopt the task of predicting the direction of the stock price Pt change over the course of short-term and long-term horizons (days) h ∈ {7D, 30D} at prediction time t.
  • Model performance is evaluated with AUC because continuous scores are more informative for investment management. statistical
    Source sentence
    We evaluate the performance on this binary classification task with AUC because the continuous scores are more informative than discrete classes for investment management applications.
  • News articles are in English and pertain to US‑based public companies. data
    Source sentence
    We curate financial news articles in English from the FNSPID Dataset (Dong et al., 2024), for US‑based public companies...
  • Transaction costs are conservatively estimated and included in the reported portfolio performance. execution
    Source sentence
    We include net performance that includes conservative estimates of the impact of transaction costs on portfolio implementation.
Show 1 more
  • The model benefits from modality‑specific routing and specialized cross‑modal pretraining. model
    Source sentence
    Our findings demonstrate that large language models can benefit significantly from structured time series context when modeling inter‑leaved sequences of text and numerical data, but only with the appropriate inductive biases.

Data

Backtest sample: 2020-01 – 2024-12 · equities

  • FNSPID Dataset · Dong et al., 2024 text-news public
    US‑based public companies
    Source sentence
    We curate financial news articles in English from the FNSPID Dataset (Dong et al., 2024), for US‑based public companies, which covers a variety of company events and news sources.

Costs

Modeled Costs are part of the reported headline results.

conservative estimates of the impact of transaction costs are included in net portfolio performance

Source sentence
We include net performance that includes conservative estimates of the impact of transaction costs on portfolio implementation.

Finds 4

Reported performance (MSE-ITT): annual return 17.0%

  • The proposed MSE‑ITT with SALMON and STW achieves the highest AUC (58.48) on the 30‑day horizon, outperforming the baseline MSE‑ITT without alignment. performance
    Source sentence
    Table 3: Results demonstrate the value of our proposed Cross‑Modal alignment objective (SALMON) with Salient‑Token Weighting (STW) compared to the baseline MSE‑ITT model finetuned without any pre‑training.
  • Modality‑specific experts reduce language model loss by 20% and improve 30‑day forecasting performance compared to shared‑parameter baselines. performance
    Source sentence
    our modality‑specific experts architecture improves both language and time series understanding capability, mitigating cross‑modal interference and allow
  • In simulated monthly long‑short portfolios, the proposed MSE‑ITT yields an annualized net return of 17.01% and a Sharpe ratio of 1.51, surpassing all unimodal and multimodal baselines. performance
    Source sentence
    Table 6: Annualized portfolio statistics of simulated investment performance
  • Zero‑shot GPT‑4o performs worse with multimodal inputs than text‑only, indicating current LLMs struggle to reason over time‑series data. negative
    Source sentence
    GPT‑4o performs better with text‑only inputs than multimodal inputs, failing to benefit from the time series context.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 22 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

https://github.com/rosskoval/mlm_text_ts — the authors' own repository

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.