{"paper_id":"2502.02199","title":"When Dimensionality Hurts: The Role of LLM Embedding Compression for Noisy Regression Tasks","authors":["Felix Drinkall","Janet B. Pierrehumbert","Stefan Zohren"],"paper_date":"2025-02-04","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2502.02199v1","page_url":"https://thequant.space/flowcharts/when-dimensionality-hurts-the-role-of-llm-embedding-compres/","scores":{"empirical_rigor":7.5,"math_complexity":4,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes a study of embedding compression for regression tasks, linking optimal dimensionality to signal‑to‑noise ratio and releasing a financial news stock returns dataset.","quote":"This paper has the following contributions: 1. Providing some evidence for a link between the optimal representational dimensionality of text in a regression task and the signal-to-noise ratio of the dataset.","summary":"Large language models (LLMs) have shown remarkable success in language modelling due to scaling laws found in model size and the hidden dimension of the model’s text representation. Yet, we demonstrate that compressed representations of text can yield better performance in LLM-based regression…","strategy":null},"assumptions":{"status":"available","items":[{"assumption":"The news content is either causing or reflecting the observed price change.","type":"model","stated":"explicit","quote":"The underlying assumption is that the news content is either causing or reflecting the observed price change."},{"assumption":"Only one target company is mentioned per sample to reduce noise.","type":"data","stated":"implied","quote":"We then apply another filter to make sure that only one of the target companies is mentioned in each sample to reduce the noise slightly."},{"assumption":"Closing bid/ask average is used as the daily price for the 50 most traded U.S. stocks.","type":"market","stated":"implied","quote":"For the 50 most traded U.S. stocks, we use the closing bid/ask average as the daily price."}]},"datasets":{"status":"available","items":[{"name":"Financial news stock returns dataset","provider":"CRSP and CommonCrawl News","kind":"text-news","access":"public","period":null,"universe":"50 most traded U.S. stocks","quote":"A financial news stock returns dataset1 released under an academic license which can be used for benchmarking purposes.","known_id":"crsp"}],"known":[{"id":"crsp","name":"CRSP","provider":"Center for Research in Security Prices","kind":"market-prices","access":"commercial","quote":"2.1 Stock Returns Dataset We combine two data sources to form the dataset: CRSP daily price data2 and news articles.","mentions":1,"used_in":1},{"id":"yahoo-finance","name":"Yahoo Finance","provider":"Yahoo","kind":"market-prices","access":"public","quote":"We source news articles via CommonCrawl News3 (Hamborg et al., 2017), scraping articles from Yahoo Finance.","mentions":1,"used_in":1}],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"daily"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"The optimal autoencoder latent dimension for the financial returns task is 8, with nearby dimensions (4, 16, 32) not significantly different.","kind":"performance","quote":"the optimal dimensionality on the financial returns task is 8, but that there is not a statistically significant difference between a dz = 8 and dz ∈{4, 16, 32}."},{"finding":"Compressing embeddings via an autoencoder mitigates overfitting and improves performance on noisy tasks such as financial return prediction.","kind":"predictive-accuracy","quote":"compressing embeddings, in a minimally supervised manner using an autoencoder’s hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction"},{"finding":"Sentiment and emotion based representations do not outperform autoencoder latent features at comparable dimensions.","kind":"performance","quote":"Both representations do not exceed the expected performance of the autoencoder features at their respective dimensions dz."},{"finding":"Extreme latent dimensions (1, 2, 128, 256, 512) lead to significant performance deterioration.","kind":"statistical","quote":"There is also a significant difference between dz = 8 and dz ∈{1, 2, 128, 256, 512}."}],"reported_performance":null,"out_of_sample":{"method":"unclear","headline_out_of_sample":null}},"independent_evidence":{"status":"available","supports":[{"paper":{"paper_id":"2403.06482","title":"Financial Default Prediction via Motif-preserving Graph Neural Network with Curriculum Learning","paper_date":"2024-03-11","authors":["Daixin Wang","Zhiqiang Zhang","Yeyu Zhao","Kai Huang","Yulin Kang","Jun Zhou"],"empirical_rigor":7.5,"page_url":"https://thequant.space/flowcharts/financial-default-prediction-via-motif-preserving-graph-neural-network-with/"},"question":"How does the size of embeddings impact model performance in graph and regression tasks?","this_paper_finding":"Extreme latent dimensions (1, 2, 128, 256, 512) lead to significant performance deterioration.","other_paper_finding":"Embedding size affects performance: too small (e.g., 32) degrades accuracy, while overly large sizes may cause overfitting.","other_paper_quote":"if the embedding size continuously increases, it may cause overfitting to the model and the accuracy may drop.","note":"Both studies report performance drops at extreme dimensions: A degrades at 32, B at 1, 2, 128, 256, 512, indicating too small or too large embeddings hurt accuracy.","confidence":0.439}],"contradicts":[],"qualifies":[],"compared_with":18,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/FelixDrinkall/financial-news-dataset","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:46:03Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}