{"paper_id":"2509.19628","title":"Multimodal Language Models with Modality-Specific Experts for Financial Forecasting from Interleaved Sequences of Text and Time Series","authors":["Ross Koval","Nicholas Andrews","Xifeng Yan"],"paper_date":"2025-09-23","paper_type":"methodological","paper_url":"https://arxiv.org/pdf/2509.19628v1","page_url":"https://thequant.space/flowcharts/multimodal-language-models-with-modality-specific-experts-fo/","scores":{"empirical_rigor":7.5,"math_complexity":6.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes a unified multimodal architecture with modality‑specific experts (MSE‑ITT) and a cross‑modal alignment framework with salient token weighting (SALMON).","quote":"We design a multimodal architecture (MSE‑ITT) that can effectively model interleaved sequences of text and time series, demonstrating state-of-the-art performance on a challenging financial forecasting task...","summary":"Text and time series data offer complementary views of financial markets: news articles provide narrative context about company events, while stock prices reflect how markets react to those events. However, despite their complementary nature, effectively integrating these interleaved modalities for…","strategy":"MSE-ITT"},"assumptions":{"status":"available","items":[{"assumption":"The forecasting task is a binary classification of stock price direction over short‑term (7D) and long‑term (30D) horizons.","type":"data","stated":"explicit","quote":"we adopt the task of predicting the direction of the stock price Pt change over the course of short-term and long-term horizons (days) h ∈ {7D, 30D} at prediction time t."},{"assumption":"Model performance is evaluated with AUC because continuous scores are more informative for investment management.","type":"statistical","stated":"explicit","quote":"We evaluate the performance on this binary classification task with AUC because the continuous scores are more informative than discrete classes for investment management applications."},{"assumption":"News articles are in English and pertain to US‑based public companies.","type":"data","stated":"explicit","quote":"We curate financial news articles in English from the FNSPID Dataset (Dong et al., 2024), for US‑based public companies..."},{"assumption":"Transaction costs are conservatively estimated and included in the reported portfolio performance.","type":"execution","stated":"explicit","quote":"We include net performance that includes conservative estimates of the impact of transaction costs on portfolio implementation."},{"assumption":"The model benefits from modality‑specific routing and specialized cross‑modal pretraining.","type":"model","stated":"explicit","quote":"Our findings demonstrate that large language models can benefit significantly from structured time series context when modeling inter‑leaved sequences of text and numerical data, but only with the appropriate inductive biases."}]},"datasets":{"status":"available","items":[{"name":"FNSPID Dataset","provider":"Dong et al., 2024","kind":"text-news","access":"public","period":null,"universe":"US‑based public companies","quote":"We curate financial news articles in English from the FNSPID Dataset (Dong et al., 2024), for US‑based public companies, which covers a variety of company events and news sources.","known_id":"fnspid"}],"known":[],"sample":{"start":"2020-01","end":"2024-12","market":null,"asset_class":"equities","frequency":null},"data_resolution":"daily"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":null,"detail":"conservative estimates of the impact of transaction costs are included in net portfolio performance","quote":"We include net performance that includes conservative estimates of the impact of transaction costs on portfolio implementation.","source":"evidence_card","rule":"R2","candidates":{"front_matter":"unclear","evidence_card":"yes","evidence_card_quoted":true},"card_detail":"conservative estimates of the impact of transaction costs are included in net portfolio performance","card_quote":"We include net performance that includes conservative estimates of the impact of transaction costs on portfolio implementation.","meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"trading-pnl","findings":[{"finding":"The proposed MSE‑ITT with SALMON and STW achieves the highest AUC (58.48) on the 30‑day horizon, outperforming the baseline MSE‑ITT without alignment.","kind":"performance","quote":"Table 3: Results demonstrate the value of our proposed Cross‑Modal alignment objective (SALMON) with Salient‑Token Weighting (STW) compared to the baseline MSE‑ITT model finetuned without any pre‑training."},{"finding":"Modality‑specific experts reduce language model loss by 20% and improve 30‑day forecasting performance compared to shared‑parameter baselines.","kind":"performance","quote":"our modality‑specific experts architecture improves both language and time series understanding capability, mitigating cross‑modal interference and allow"},{"finding":"In simulated monthly long‑short portfolios, the proposed MSE‑ITT yields an annualized net return of 17.01% and a Sharpe ratio of 1.51, surpassing all unimodal and multimodal baselines.","kind":"performance","quote":"Table 6: Annualized portfolio statistics of simulated investment performance"},{"finding":"Zero‑shot GPT‑4o performs worse with multimodal inputs than text‑only, indicating current LLMs struggle to reason over time‑series data.","kind":"negative","quote":"GPT‑4o performs better with text‑only inputs than multimodal inputs, failing to benefit from the time series context."}],"reported_performance":{"strategy":"MSE-ITT","sharpe":null,"sharpe_benchmark":null,"annual_return_pct":17.01,"total_return_pct":null,"max_drawdown_pct":null},"out_of_sample":{"method":"holdout","headline_out_of_sample":true}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":22,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/rosskoval/mlm_text_ts","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:26:27Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}