Paper: arXiv 2511.08608
Authors: Rakeshkumar H Sodha
Abstract
Problem. “Thinking” LLMs (TLLMs) expose explicit or hidden reasoning traces and are widely believed to generalize better on complex tasks than direct LLMs. Whether this promise carries to noisy, heavy-tailed and regime-switching financial data remains unclear. Approach. Using Indian equities (NIFTY constituents), we run a rolling 48m/1m walk-forward evaluation at horizon k = 1 day and dial cross-sectional complexity via the universe size U in {“5, 11, 21, 36”} while keeping the reasoning budget fixed (B = 512 tokens) for the TLLM. We compare a direct LLM (gpt-4o-mini), a TLLM (gpt-5), and classical learners (ridge, random forest) on cross-sectional ranking loss 1 - IC, MSE, and long/short backtests with realistic costs. Statistical confidence is measured with Diebold-Mariano, Pesaran-Timmermann, and SPA tests. Main findings. (i) As U grows under a fixed budget B, the TLLM’s ranking quality deteriorates, whereas the direct LLM remains flat and classical baselines are stable. (ii) TLLM variance is higher, requiring ex-post calibration (winsorization and blending) for stability. (iii) Portfolio results under transaction costs do not support a net advantage for the TLLM. Hypotheses. Our results are consistent with the following testable hypotheses: H1 (Capacity-Complexity Mismatch): for fixed B, TLLM accuracy degrades superlinearly in cross-sectional complexity. H2 (Reasoning Variance): TLLM outputs exhibit higher dispersion date-by-date than direct LLMs, increasing error bars and turnover. H3 (Domain Misfit): next-token prediction objectives and token-budgeted inference are poorly aligned with heavy-tailed, weakly predictable stock returns. Implication. In our setting, “thinking” LLMs are not yet ready to replace classical or direct methods for short-horizon stock ranking; scaling the reasoning budget and/or re-aligning objectives appears necessary.
Complexity vs Empirical Score
- Math Complexity: 6.5/10
- Empirical Rigor: 8.0/10
- Quadrant: Holy Grail — high math complexity, high empirical rigor
Why this score: The paper employs advanced statistical metrics (DM, PT, SPA tests) and structured hypothesis testing (H1-H3), indicating moderate-to-high math complexity, and features a rigorous, multi-faceted empirical evaluation with walk-forward backtests, realistic transaction costs, and robustness checks, aligning it with the Holy Grail quadrant.
Research Flowchart
flowchart TD
A["Research Goal<br>Test TLLMs vs. Direct LLMs on<br>noisy financial data"] --> B["Methodology: Rolling Walk-Forward"]
B --> C["Data: NIFTY Constituents<br>Complexity U = {'5, 11, 21, 36}"]
C --> D["Models & Process<br>TLLM (gpt-5) vs. Direct LLM (gpt-4o-mini)<br>vs. Classical (Ridge, RF)<br>Fixed Budget B = 512 tokens"]
D --> E{"Evaluation Metrics"}
E --> F["Ranking Loss 1-IC<br>MSE<br>Long/Short Backtests"]
E --> G["Statistical Tests<br>Diebold-Mariano, SPA"]
F --> H["Key Findings"]
G --> H
H --> I(("1. TLLM accuracy drops as U grows"))
H --> J(("2. TLLM variance higher<br>requires calibration"))
H --> K(("3. No net advantage under costs"))
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a rolling walk‑forward evaluation comparing a direct LLM, a “thinking” LLM, and classical learners on short‑horizon stock ranking with realistic transaction costs.
Source sentence
We compare a direct LLM (gpt-4o-mini), a TLLM (gpt-5), and classical learners (ridge, random forest) on cross‑sectional ranking loss 1 −IC, MSE, and long/short backtests with realistic costs.
Assumes 6
- Short‑horizon stock ranking at a 1‑day horizon on NIFTY constituents. market
Source sentence
Our conclusions are conditional on: (i) short-horizon stock ranking at k=1 day on NIFTY constituents
- Cross‑sectional complexity is varied by the universe size U ∈{5,11,21,36}. data
Source sentence
cross‑sectional complexity dialed via U ∈{5, 11, 21, 36}
- The reasoning budget for the TLLM is fixed at B = 512 tokens. model
Source sentence
while keeping the reasoning budget fixed (B = 512 tokens) for the TLLM
- Feature blocks and per‑date calibration (z‑scoring, winsorization) are applied as described. execution
Source sentence
Per‑date calibration: cross‑sectional z‑scoring with safe fallback when σ(d) ≈0; rescale to training target mean/std and winsorize (5–95%) per date.
Show 2 more
- The sample covers the period 2019 to the present using a 48‑month train / 1‑month test walk‑forward protocol. data
Source sentence
the 2019–present sample under our walk‑forward protocol
- Realistic one‑way transaction costs, borrow costs, and turnover caps are applied to daily rebalances. execution
Source sentence
realistic one‑way transaction costs, borrow on shorts, and turnover caps applied to daily rebalances
Data
Indian equities (NIFTY constituents) · equities · daily
- NIFTY constituents market-prices unknown
Source sentence
We evaluate the liquid Indian equity universe (NIFTY constituents).
Costs
Modeled
Includes slippage, brokerage, STT, other fees, and borrow costs; one‑way cost denoted cone‑way in basis points with turnover caps; robustness checked by ±50% perturbation.
Source sentence
We incorporate slippage, brokerage, STT, other fees, and borrow costs consistent with the Indian market microstructure.
Finds 4
Sharpe -0.43 (benchmark 4.16)
- With a fixed budget B=512, the TLLM’s ranking loss (1‑IC) worsens monotonically as universe size U grows, while the direct LLM stays flat and classical baselines remain stable. predictive-accuracy
Source sentence
With fixed TLLM budget B = 512, ranking loss (1–IC) increases monotonically as the universe size U increases, while the direct LLM remains comparatively flat and classical baselines are competitive and stable.
- TLLM outputs exhibit higher variance date‑by‑date, requiring ex‑post calibration (winsorization and blending) for stability. statistical
Source sentence
TLLM variance is higher, requiring ex‑post calibration (winsorization and blending) for stability.
- Portfolio results under realistic transaction costs do not provide a net advantage for the TLLM. performance
Source sentence
Portfolio results under transaction costs do not support a net advantage for the TLLM.
- Cost‑adjusted net Sharpe ranking places ridge regression first, calibrated direct LLM second, and calibrated TLLM (blended) below both, with raw TLLM showing negative Sharpe. performance
Source sentence
ridge ranks highest on net Sharpe, followed by the calibrated direct LLM; the calibrated‑and‑blended TLLM improves upon raw TLLM but remains below the baselines.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 45 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.