Paper: arXiv 2505.01575

Authors: Shanyan Lai

Abstract

This paper proposes an innovative Transformer model, Single-directional representative from Transformer (SERT), for US large capital stock pricing. It also innovatively applies the pre-trained Transformer models under the stock pricing and factor investment context. They are compared with standard Transformer models and encoder-only Transformer models in three periods covering the entire COVID-19 pandemic to examine the model adaptivity and suitability during the extreme market fluctuations. Namely, pre-COVID-19 period (mild up-trend), COVID-19 period (sharp up-trend with deep down shock) and 1-year post-COVID-19 (high fluctuation sideways movement). The best proposed SERT model achieves the highest out-of-sample R2, 11.2% and 10.91% respectively, when extreme market fluctuation takes place followed by pre-trained Transformer models (10.38% and 9.15%). Their Trend-following-based strategy wise performance also proves their excellent capability for hedging downside risks during market shocks. The proposed SERT model achieves a Sortino ratio 47% higher than the buy-and-hold benchmark in the equal-weighted portfolio and 28% higher in the value-weighted portfolio when the pandemic period is attended. It proves that Transformer models have a great capability to capture patterns of temporal sparsity data in the asset pricing factor model, especially with considerable volatilities. We also find the softmax signal filter as the common configuration of Transformer models in alternative contexts, which only eliminates differences between models, but does not improve strategy-wise performance, while increasing attention heads improve the model performance insignificantly and applying the ’layer norm first’ method do not boost the model performance in our case.

Complexity vs Empirical Score

  • Math Complexity: 7.5/10
  • Empirical Rigor: 6.0/10 (capped from 8.0: transaction costs not modelled; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 8.0)

Why this score: The paper involves advanced neural network architectures (Transformers, attention mechanisms, pre-training) with mathematical formulations and a novel model (SERT), indicating high math complexity. It is backed by specific empirical evaluation across multiple market regimes (pre-COVID, COVID, post-COVID), out-of-sample metrics (R², Sortino ratios), and portfolio strategy comparisons, demonstrating high empirical rigor.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Asset Pricing with Transformers"] --> B["Data Collection<br>US Large Cap Stocks, Factor Data"]
  B --> C["Methodology<br>Develop & Compare SERT vs Standard Transformers"]
  C --> D["Model Training<br>Pre-training & Fine-tuning over 3 Market Periods"]
  D --> E["Evaluation & Strategy<br>Out-of-Sample R2 & Trend-Following Portfolio Performance"]
  E --> F{"Key Outcomes"}
  F --> F1["SERT achieves highest R2<br>(11.2% / 10.91%) in volatile periods"]
  F --> F2["Superior hedging & Sortino Ratio<br>(47% / 28% higher) during COVID"]
  F --> F3["Softmax filter & Layer Norm<br>did not boost performance"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a new Transformer model called Single-directional representative from Transformer (SERT) for pricing US large-cap stocks.

Source sentence
This paper proposes an innovative Transformer model, Single-directional representative from Transformer (SERT), for US large capital stock pricing.

Assumes 5

  • Stocks are selected from the top 15% of market capitalisation and have full data in the testing period with less than 50% missing values during training. data
    Source sentence
    They satisfy the conditions of having full available data in the testing period and less than 50% of values missing during the training period for meeting the requirement of the ‘going concern’ and data representability.
  • The selected stocks are assumed to be ‘too‑big‑to‑fail’ and thus represent a realistic investment universe. market
    Source sentence
    As a benefit of the assumption of ‘too-big-to-fail’ and ‘going concern’, this setting is more realistic for practitioners since it protects the investors from systematic risks such as companies are delisted unexpectedly due to insolvency.
  • Risk‑free rate is taken from the Kenneth R. French Data Library for constructing excess returns and evaluating Sharpe and Sortino ratios. data
    Source sentence
    The risk-free rate for constructing the excess returns and evaluating the Sharp ratio and Sortino ratio is derived from the Kenneth R. French Data Library.
  • The softmax trading signal filter removes the worst 50% of signals each month for 420 stocks. execution
    Source sentence
    The softmax trading signal filter utilizes the softmax function on sign signals and filters out the worst 50% of trading signals for 420 stocks each month.
Show 1 more
  • Model performance is evaluated using out‑of‑sample R² and Sortino ratio as primary metrics. statistical
    Source sentence
    The best proposed SERT model achieves the highest out-of-sample R2, 11.2% and 10.91% respectively

Data

Backtest sample: 1957-01 – 2022-12 · US large-cap stocks · equities

  • Monthly sorted portfolio returns as factors · Andrew Y.; Zimmermann (reconstructed) other unknown
    Source sentence
    In this research, the monthly sorted portfolio returns as factors reconstructed by Andrew Y.; Zimmermann [52] are deployed as the observable factors (original input).
  • 420 large‑capital US stocks excess returns · CRSP via Wharton Research Data Services (WRDS) market-prices commercial
    January 1957 to December 20… · US NASDAQ and NYSE stocks filtered from the top 15% market cap
    Source sentence
    the 420 large capital US stocks’ excess returns as the labels (original output). These stocks come from the US NASDAQ and NYSE markets which are derived from CRSP of Wharton Research Data Services (WRDS).

Costs

Not modeled A trading or execution result is reported without costs.

Finds 4

Reported performance (Trans3 (pre-trained Transformer)): Sharpe 0.31 · annual return 8.0% · max drawdown 32.6%

  • The SERT model attains the highest out‑of‑sample R² of 11.2% (and 10.91%) during extreme market fluctuations, outperforming standard and encoder‑only Transformers. predictive-accuracy
    Source sentence
    The best proposed SERT model achieves the highest out-of-sample R2, 11.2% and 10.91% respectively, when extreme market fluctuation takes place followed by pre-trained Transformer models (10.38% and 9.15%).
  • SERT’s Sortino ratio is 47% higher than the buy‑and‑hold benchmark for equal‑weighted portfolios and 28% higher for value‑weighted portfolios during the pandemic period. performance
    Source sentence
    The proposed SERT model achieves a Sortino ratio 47% higher than the buy-and-hold benchmark in the equal-weighted portfolio and 28% higher in the value-weighted portfolio when the pandemic period is attended.
  • Pre‑trained Transformer models significantly outperform buy‑and‑hold benchmarks during extreme market periods (2112 and 2212) according to Sortino ratio. performance
    Source sentence
    the proposed pretrained Transformer models significantly outperform the benchmark models as well as the buy-and-hold strategy (BHE) during the extreme market fluctuation (2112 and 2212) according to the Sortino ratio.
  • Increasing attention‑head numbers improves model performance only insignificantly; layer‑norm‑first does not boost performance. negative
    Source sentence
    while increasing attention heads improve the model performance insignificantly and applying the 'layer norm first' method do not boost the model performance in our case.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 25 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 2 papers in the archive

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.