{"paper_id":"2507.16548","title":"Alternative Loss Function in Evaluation of Transformer Models","authors":["Jakub Michańków","Paweł Sakowski","Robert Ślepaczuk"],"paper_date":"2025-07-22","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2507.16548v2","page_url":"https://thequant.space/flowcharts/alternative-loss-function-in-evaluation-of-transformer-model/","scores":{"empirical_rigor":6,"math_complexity":6,"quadrant":"Holy Grail"},"proposes":{"status":"available","statement":"The paper proposes applying the Mean Absolute Directional Loss (MADL) function to train transformer and LSTM models for generating trading signals and demonstrates the transformer’s superiority.","quote":"Therefore, in this research, through empirical experiments on equity and cryptocurrency assets, we apply the Mean Absolute Directional Loss (MADL) function, which is more adequate for optimizing forecast-generating models used in algorithmic investment strategies.","summary":"The proper design and architecture of testing machine learning models, especially in their application to quantitative finance problems, is crucial. The most important aspect of this process is selecting an adequate loss function for training, validation, estimation purposes, and hyperparameter…","strategy":null},"assumptions":{"status":"none_found","items":[]},"datasets":{"status":"available","items":[{"name":"Daily data from six assets (Bitcoin, Ethereum, Litecoin, JP Morgan, S&P 500, Exxon Mobil)","provider":null,"kind":"market-prices","access":"unknown","period":"out-of-sample period extended over eight","universe":"cryptocurrencies and equity stocks","quote":"The models were applied to daily data from six assets (cryptocurrencies, including Bitcoin, Ethereum, and Litecoin, and equity stocks, including JP Morgan, S&P 500, and Exxon Mobil). The walk-forward procedure was used to include the out-of-sample period, which extended over eight years."}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"daily"},"transaction_costs":{"status":"available","modeled":"no","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R1","candidates":{"front_matter":"no","evidence_card":"unclear","evidence_card_quoted":false},"meaning":"a trading or execution result is reported without costs"},"results":{"status":"available","evaluation_type":"trading-pnl","findings":[{"finding":"Transformer models produce higher risk‑adjusted returns than both LSTM and Buy&Hold strategies for all six assets.","kind":"performance","quote":"The transformer outperforms the Buy&Hold and LSTM-based strategies for both types of assets under investigation. This architecture produces higher risk-adjusted returns compared with both the LSTM and the Buy&Hold strategy."},{"finding":"In almost every case, transformer results are significantly better than those obtained with LSTM.","kind":"performance","quote":"we show that in almost every case, Transformer results are significantly better than those obtained with LSTM."}],"reported_performance":null,"out_of_sample":{"method":"walk-forward","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":18,"citations":{"cited_by_in_corpus":1,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":1},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":false,"code_url":null,"note":"No repository presented by the authors as their own was found in the paper. Absence here is not proof that no code exists."},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:31:30Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}