Paper: arXiv 2502.02199

Authors: Felix Drinkall, Janet B. Pierrehumbert, Stefan Zohren

Abstract

Large language models (LLMs) have shown remarkable success in language modelling due to scaling laws found in model size and the hidden dimension of the model’s text representation. Yet, we demonstrate that compressed representations of text can yield better performance in LLM-based regression tasks. In this paper, we compare the relative performance of embedding compression in three different signal-to-noise contexts: financial return prediction, writing quality assessment and review scoring. Our results show that compressing embeddings, in a minimally supervised manner using an autoencoder’s hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction; but that compression reduces performance on tasks that have high causal dependencies between the input and target data. Our results suggest that the success of interpretable compressed representations such as sentiment may be due to a regularising effect.

Complexity vs Empirical Score

  • Math Complexity: 4.0/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper employs standard machine learning techniques (autoencoders, regression) with minimal advanced mathematical derivations, but demonstrates strong empirical rigor through the use of three distinct real-world datasets (including financial news and stock returns), specific model architectures (all-mpnet-base-v2), and clear performance benchmarking, positioning it as a practical, data-heavy investigation.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Does LLM embedding compression<br>improve regression performance?"]
  B["Dataset & Signal-to-Noise Contexts<br>Financial (Low SNR)<br>Writing Quality (High SNR)<br>Review Scoring (Medium SNR)"]
  C["Methodology<br>Train Autoencoder on LLM Embeddings<br>Compress to latent space representation"]
  D["Computation<br>Regression Task: Predict Target<br>Compare Raw vs. Compressed Embeddings"]
  E{"Key Findings & Outcomes"}
  E --> F["Low SNR (Financial)<br>Compression improves performance<br>Reduces overfitting"]
  E --> G["High SNR (Writing/Review)<br>Compression reduces performance<br>Removes causal dependencies"]
  E --> H["Interpretability<br>Compressed features act as<br>effective regularization"]

  A --> B
  B --> C
  C --> D
  D --> E

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a study of embedding compression for regression tasks, linking optimal dimensionality to signal‑to‑noise ratio and releasing a financial news stock returns dataset.

Source sentence
This paper has the following contributions: 1. Providing some evidence for a link between the optimal representational dimensionality of text in a regression task and the signal-to-noise ratio of the dataset.

Assumes 3

  • The news content is either causing or reflecting the observed price change. model
    Source sentence
    The underlying assumption is that the news content is either causing or reflecting the observed price change.
  • Only one target company is mentioned per sample to reduce noise. data
    Source sentence
    We then apply another filter to make sure that only one of the target companies is mentioned in each sample to reduce the noise slightly.
  • Closing bid/ask average is used as the daily price for the 50 most traded U.S. stocks. market
    Source sentence
    For the 50 most traded U.S. stocks, we use the closing bid/ask average as the daily price.

Data

  • Financial news stock returns dataset · CRSP and CommonCrawl News text-news public
    50 most traded U.S. stocks
    Source sentence
    A financial news stock returns dataset1 released under an academic license which can be used for benchmarking purposes.

Recognised sources: CRSP Yahoo Finance

Costs

Not applicable The paper reports no trading or execution result.

Finds 4

  • The optimal autoencoder latent dimension for the financial returns task is 8, with nearby dimensions (4, 16, 32) not significantly different. performance
    Source sentence
    the optimal dimensionality on the financial returns task is 8, but that there is not a statistically significant difference between a dz = 8 and dz ∈{4, 16, 32}.
  • Compressing embeddings via an autoencoder mitigates overfitting and improves performance on noisy tasks such as financial return prediction. predictive-accuracy
    Source sentence
    compressing embeddings, in a minimally supervised manner using an autoencoder’s hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction
  • Sentiment and emotion based representations do not outperform autoencoder latent features at comparable dimensions. performance
    Source sentence
    Both representations do not exceed the expected performance of the autoencoder features at their respective dimensions dz.
  • Extreme latent dimensions (1, 2, 128, 256, 512) lead to significant performance deterioration. statistical
    Source sentence
    There is also a significant difference between dz = 8 and dz ∈{1, 2, 128, 256, 512}.

Checked by others

Supported by

  • Financial Default Prediction via Motif-preserving Graph Neural Network with Curriculum Learning (2024)
    On the question: How does the size of embeddings impact model performance in graph and regression tasks?
    Compare the findings

    This paper: Extreme latent dimensions (1, 2, 128, 256, 512) lead to significant performance deterioration.

    That paper: Embedding size affects performance: too small (e.g., 32) degrades accuracy, while overly large sizes may cause overfitting.

    Both studies report performance drops at extreme dimensions: A degrades at 32, B at 1, 2, 128, 256, 512, indicating too small or too large embeddings hurt accuracy.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (56)

  1. The Theory of Parsing, Translation and Compiling (1972)
  2. Dissociating Language and Thought in Large Language Models (2024) — link
  3. Re (Visiting) Large Language Models in Finance (2024) — link
  4. Graph-enhanced Large Language Models in Asynchronous Plan Reasoning (2025) — link
  5. Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024) — link
  6. Age Related Differences in Language Usage and Reading between E nglish Monolinguals and Bilinguals (2013) — link
  7. Challenges and Strategies in Cross-Cultural NLP (2022) — link
  8. Geographic Adaptation of Pretrained Language Models (2024) — link
and 48 more
  1. Bridging Language and Items for Retrieval and Recommendation (2024) — link
  2. Feedback Prize - English Language Learning (2022) — link
  3. Software Applications User Reviews (2017) — link
  4. Efficient Prompting Methods for Large Language Models: A Survey (2024) — link
  5. Predicting Stock Returns in the Presence of Uncertain Structural Changes and Sample Noise (2017) — link
  6. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024) — link
  7. Macroeconomic Forecasting Through News, Emotions and Narrative (2021) — link
  8. Automatic Detection of Relevant Information, Predictions and Forecasts in Financial News Through Topic Modelling with Latent Dirichlet Allocation — → 2404.01338
  9. Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data (2024) — link
  10. Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data (2023) — link
  11. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2022) — link
  12. Training Compute-optimal Large Language Models (2022) — link
  13. Regression or Classification? Automated Essay Scoring for N orwegian (2019) — link
  14. The Probable Error of a Mean (1908) — link
  15. A Comprehensive Survey on Regularization Strategies in Machine Learning (2022) — link
  16. To Repeat or Not To Repeat: Insights from Scaling LLM Under Token-crisis (2023) — link
  17. Random Forests (2001) — link
  18. Robustness of Random Forests for Regression (2012) — link
  19. news-please: A Generic News Crawler and Extractor (2017) — link
  20. Emotion English DistilRoBERTa-base (2022)
  21. DistilBERT: A Distilled Version of BERT—Smaller, Faster, Cheaper, and Lighter (2019) — link
  22. Understanding LLM Embeddings for Regression (2024) — link
  23. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024) — link
  24. Are Decoder-Only Language Models Better than Encoder-Only Language Models in Understanding Word Meaning? (2024) — link
  25. Noise (1986) — link
  26. Is All That Talk Just Noise? The Information Content of Internet Stock Message Boards (2004) — link
  27. Deep Attentive Learning for Stock Movement Prediction From Social Media Text and Company Correlations (2020) — link
  28. Forecasting COVID -19 Caseloads Using Unsupervised Embedding Clusters of Social Media Posts (2022) — link
  29. Scaling Laws for Neural Language Models (2020) — link
  30. Sentence- BERT : Sentence Embeddings using S iamese BERT -Networks (2019) — link
  31. MPNet: Masked and Permuted Pre-training for Language Understanding (2020) — link
  32. MTEB: Massive Text Embedding Benchmark (2023) — link
  33. Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs — → 2407.17624
  34. The Llama 3 Herd of Models (2024) — link
  35. Introduction to the C o NLL -2003 Shared Task: Language-Independent Named Entity Recognition (2003) — link
  36. BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
  37. From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise Through Online Reviews (2013) — link
  38. Robust Estimation of a Location Parameter (1964) — link
  39. Principal Component Analysis (1987) — link
  40. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction (2018) — link
  41. Visualizing Data using t-SNE (2008) — link
  42. Robust Estimation of a Location Parameter (1992) — link
  43. Publications Manual (1983)
  44. Character-level Convolutional Networks for Text Classification (2015) — link
  45. Alternation (1981) — link
  46. Scalable training of L1 -regularized log-linear models (2007)
  47. Algorithms on Strings, Trees and Sequences (1997)
  48. Yara Parser: A Fast and Accurate Dependency Parser (2015) — link

2 of 56 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.