{"paper_id":"2511.08608","title":"When Reasoning Fails: Evaluating 'Thinking' LLMs for Stock Prediction","authors":["Rakeshkumar H Sodha"],"paper_date":"2025-11-05","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2511.08608v1","page_url":"https://thequant.space/flowcharts/when-reasoning-fails-evaluating-thinking-llms-for-stock-p/","scores":{"empirical_rigor":8,"math_complexity":6.5,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes a rolling walk‑forward evaluation comparing a direct LLM, a “thinking” LLM, and classical learners on short‑horizon stock ranking with realistic transaction costs.","quote":"We compare a direct LLM (gpt-4o-mini), a TLLM (gpt-5), and classical learners (ridge, random forest) on cross‑sectional ranking loss 1 −IC, MSE, and long/short backtests with realistic costs.","summary":"Problem. “Thinking” LLMs (TLLMs) expose explicit or hidden reasoning traces and are widely believed to generalize better on complex tasks than direct LLMs. Whether this promise carries to noisy, heavy-tailed and regime-switching financial data remains unclear. Approach. Using Indian equities (NIFTY…","strategy":"Thinking LLM (gpt-5) with B=512"},"assumptions":{"status":"available","items":[{"assumption":"Short‑horizon stock ranking at a 1‑day horizon on NIFTY constituents.","type":"market","stated":"explicit","quote":"Our conclusions are conditional on: (i) short-horizon stock ranking at k=1 day on NIFTY constituents"},{"assumption":"Cross‑sectional complexity is varied by the universe size U ∈{5,11,21,36}.","type":"data","stated":"explicit","quote":"cross‑sectional complexity dialed via U ∈{5, 11, 21, 36}"},{"assumption":"The reasoning budget for the TLLM is fixed at B = 512 tokens.","type":"model","stated":"explicit","quote":"while keeping the reasoning budget fixed (B = 512 tokens) for the TLLM"},{"assumption":"Feature blocks and per‑date calibration (z‑scoring, winsorization) are applied as described.","type":"execution","stated":"explicit","quote":"Per‑date calibration: cross‑sectional z‑scoring with safe fallback when σ(d) ≈0; rescale to training target mean/std and winsorize (5–95%) per date."},{"assumption":"The sample covers the period 2019 to the present using a 48‑month train / 1‑month test walk‑forward protocol.","type":"data","stated":"explicit","quote":"the 2019–present sample under our walk‑forward protocol"},{"assumption":"Realistic one‑way transaction costs, borrow costs, and turnover caps are applied to daily rebalances.","type":"execution","stated":"explicit","quote":"realistic one‑way transaction costs, borrow on shorts, and turnover caps applied to daily rebalances"}]},"datasets":{"status":"available","items":[{"name":"NIFTY constituents","provider":null,"kind":"market-prices","access":"unknown","period":"2019‑present","universe":"Indian equity universe (NIFTY constituents)","quote":"We evaluate the liquid Indian equity universe (NIFTY constituents)."}],"known":[],"sample":{"start":null,"end":null,"market":"Indian equities (NIFTY constituents)","asset_class":"equities","frequency":"daily"},"data_resolution":"daily"},"transaction_costs":{"status":"available","modeled":"yes","headline_net_of_costs":true,"detail":"Includes slippage, brokerage, STT, other fees, and borrow costs; one‑way cost denoted cone‑way in basis points with turnover caps; robustness checked by ±50% perturbation.","quote":"We incorporate slippage, brokerage, STT, other fees, and borrow costs consistent with the Indian market microstructure.","source":"front_matter","rule":"R3","candidates":{"front_matter":"yes","evidence_card":"yes","evidence_card_quoted":true},"card_detail":"Includes slippage, brokerage, STT, other fees, and borrow costs; one‑way cost denoted cone‑way in basis points with turnover caps; robustness checked by ±50% perturbation.","card_quote":"We incorporate slippage, brokerage, STT, other fees, and borrow costs consistent with the Indian market microstructure.","meaning":"costs are part of the reported headline results"},"results":{"status":"available","evaluation_type":"unclear","findings":[{"finding":"With a fixed budget B=512, the TLLM’s ranking loss (1‑IC) worsens monotonically as universe size U grows, while the direct LLM stays flat and classical baselines remain stable.","kind":"predictive-accuracy","quote":"With fixed TLLM budget B = 512, ranking loss (1–IC) increases monotonically as the universe size U increases, while the direct LLM remains comparatively flat and classical baselines are competitive and stable."},{"finding":"TLLM outputs exhibit higher variance date‑by‑date, requiring ex‑post calibration (winsorization and blending) for stability.","kind":"statistical","quote":"TLLM variance is higher, requiring ex‑post calibration (winsorization and blending) for stability."},{"finding":"Portfolio results under realistic transaction costs do not provide a net advantage for the TLLM.","kind":"performance","quote":"Portfolio results under transaction costs do not support a net advantage for the TLLM."},{"finding":"Cost‑adjusted net Sharpe ranking places ridge regression first, calibrated direct LLM second, and calibrated TLLM (blended) below both, with raw TLLM showing negative Sharpe.","kind":"performance","quote":"ridge ranks highest on net Sharpe, followed by the calibrated direct LLM; the calibrated‑and‑blended TLLM improves upon raw TLLM but remains below the baselines."}],"reported_performance":{"strategy":"Thinking LLM (gpt-5) with B=512","sharpe":-0.426,"sharpe_benchmark":4.156,"annual_return_pct":null,"total_return_pct":null,"max_drawdown_pct":null},"out_of_sample":{"method":"walk-forward","headline_out_of_sample":true}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":45,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:22:45Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}