Paper: arXiv 2502.02199
Authors: Felix Drinkall, Janet B. Pierrehumbert, Stefan Zohren
Abstract
Large language models (LLMs) have shown remarkable success in language modelling due to scaling laws found in model size and the hidden dimension of the model’s text representation. Yet, we demonstrate that compressed representations of text can yield better performance in LLM-based regression tasks. In this paper, we compare the relative performance of embedding compression in three different signal-to-noise contexts: financial return prediction, writing quality assessment and review scoring. Our results show that compressing embeddings, in a minimally supervised manner using an autoencoder’s hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction; but that compression reduces performance on tasks that have high causal dependencies between the input and target data. Our results suggest that the success of interpretable compressed representations such as sentiment may be due to a regularising effect.
Complexity vs Empirical Score
- Math Complexity: 4.0/10
- Empirical Rigor: 7.5/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper employs standard machine learning techniques (autoencoders, regression) with minimal advanced mathematical derivations, but demonstrates strong empirical rigor through the use of three distinct real-world datasets (including financial news and stock returns), specific model architectures (all-mpnet-base-v2), and clear performance benchmarking, positioning it as a practical, data-heavy investigation.
Research Flowchart
flowchart TD
A["Research Goal<br>Does LLM embedding compression<br>improve regression performance?"]
B["Dataset & Signal-to-Noise Contexts<br>Financial (Low SNR)<br>Writing Quality (High SNR)<br>Review Scoring (Medium SNR)"]
C["Methodology<br>Train Autoencoder on LLM Embeddings<br>Compress to latent space representation"]
D["Computation<br>Regression Task: Predict Target<br>Compare Raw vs. Compressed Embeddings"]
E{"Key Findings & Outcomes"}
E --> F["Low SNR (Financial)<br>Compression improves performance<br>Reduces overfitting"]
E --> G["High SNR (Writing/Review)<br>Compression reduces performance<br>Removes causal dependencies"]
E --> H["Interpretability<br>Compressed features act as<br>effective regularization"]
A --> B
B --> C
C --> D
D --> E
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a study of embedding compression for regression tasks, linking optimal dimensionality to signal‑to‑noise ratio and releasing a financial news stock returns dataset.
Source sentence
This paper has the following contributions: 1. Providing some evidence for a link between the optimal representational dimensionality of text in a regression task and the signal-to-noise ratio of the dataset.
Assumes 3
- The news content is either causing or reflecting the observed price change. model
Source sentence
The underlying assumption is that the news content is either causing or reflecting the observed price change.
- Only one target company is mentioned per sample to reduce noise. data
Source sentence
We then apply another filter to make sure that only one of the target companies is mentioned in each sample to reduce the noise slightly.
- Closing bid/ask average is used as the daily price for the 50 most traded U.S. stocks. market
Source sentence
For the 50 most traded U.S. stocks, we use the closing bid/ask average as the daily price.
Data
- Financial news stock returns dataset text-news public
Source sentence
A financial news stock returns dataset1 released under an academic license which can be used for benchmarking purposes.
CRSP Yahoo Finance
Costs
Not applicable
Finds 4
- The optimal autoencoder latent dimension for the financial returns task is 8, with nearby dimensions (4, 16, 32) not significantly different. performance
Source sentence
the optimal dimensionality on the financial returns task is 8, but that there is not a statistically significant difference between a dz = 8 and dz ∈{4, 16, 32}.
- Compressing embeddings via an autoencoder mitigates overfitting and improves performance on noisy tasks such as financial return prediction. predictive-accuracy
Source sentence
compressing embeddings, in a minimally supervised manner using an autoencoder’s hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction
- Sentiment and emotion based representations do not outperform autoencoder latent features at comparable dimensions. performance
Source sentence
Both representations do not exceed the expected performance of the autoencoder features at their respective dimensions dz.
- Extreme latent dimensions (1, 2, 128, 256, 512) lead to significant performance deterioration. statistical
Source sentence
There is also a significant difference between dz = 8 and dz ∈{1, 2, 128, 256, 512}.
Checked by others
Supported by
- Financial Default Prediction via Motif-preserving Graph Neural Network with Curriculum Learning
Compare the findings
This paper: Extreme latent dimensions (1, 2, 128, 256, 512) lead to significant performance deterioration.
That paper: Embedding size affects performance: too small (e.g., 32) degrades accuracy, while overly large sizes may cause overfitting.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (56)
- The Theory of Parsing, Translation and Compiling (1972)
- Dissociating Language and Thought in Large Language Models (2024) — link
- Re (Visiting) Large Language Models in Finance (2024) — link
- Graph-enhanced Large Language Models in Asynchronous Plan Reasoning (2025) — link
- Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency (2024) — link
- Age Related Differences in Language Usage and Reading between E nglish Monolinguals and Bilinguals (2013) — link
- Challenges and Strategies in Cross-Cultural NLP (2022) — link
- Geographic Adaptation of Pretrained Language Models (2024) — link
and 48 more
- Bridging Language and Items for Retrieval and Recommendation (2024) — link
- Feedback Prize - English Language Learning (2022) — link
- Software Applications User Reviews (2017) — link
- Efficient Prompting Methods for Large Language Models: A Survey (2024) — link
- Predicting Stock Returns in the Presence of Uncertain Structural Changes and Sample Noise (2017) — link
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024) — link
- Macroeconomic Forecasting Through News, Emotions and Narrative (2021) — link
- Automatic Detection of Relevant Information, Predictions and Forecasts in Financial News Through Topic Modelling with Latent Dirichlet Allocation — → 2404.01338
- Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data (2024) — link
- Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data (2023) — link
- AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2022) — link
- Training Compute-optimal Large Language Models (2022) — link
- Regression or Classification? Automated Essay Scoring for N orwegian (2019) — link
- The Probable Error of a Mean (1908) — link
- A Comprehensive Survey on Regularization Strategies in Machine Learning (2022) — link
- To Repeat or Not To Repeat: Insights from Scaling LLM Under Token-crisis (2023) — link
- Random Forests (2001) — link
- Robustness of Random Forests for Regression (2012) — link
- news-please: A Generic News Crawler and Extractor (2017) — link
- Emotion English DistilRoBERTa-base (2022)
- DistilBERT: A Distilled Version of BERT—Smaller, Faster, Cheaper, and Lighter (2019) — link
- Understanding LLM Embeddings for Regression (2024) — link
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024) — link
- Are Decoder-Only Language Models Better than Encoder-Only Language Models in Understanding Word Meaning? (2024) — link
- Noise (1986) — link
- Is All That Talk Just Noise? The Information Content of Internet Stock Message Boards (2004) — link
- Deep Attentive Learning for Stock Movement Prediction From Social Media Text and Company Correlations (2020) — link
- Forecasting COVID -19 Caseloads Using Unsupervised Embedding Clusters of Social Media Posts (2022) — link
- Scaling Laws for Neural Language Models (2020) — link
- Sentence- BERT : Sentence Embeddings using S iamese BERT -Networks (2019) — link
- MPNet: Masked and Permuted Pre-training for Language Understanding (2020) — link
- MTEB: Massive Text Embedding Benchmark (2023) — link
- Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMs — → 2407.17624
- The Llama 3 Herd of Models (2024) — link
- Introduction to the C o NLL -2003 Shared Task: Language-Independent Named Entity Recognition (2003) — link
- BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
- From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise Through Online Reviews (2013) — link
- Robust Estimation of a Location Parameter (1964) — link
- Principal Component Analysis (1987) — link
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction (2018) — link
- Visualizing Data using t-SNE (2008) — link
- Robust Estimation of a Location Parameter (1992) — link
- Publications Manual (1983)
- Character-level Convolutional Networks for Text Classification (2015) — link
- Alternation (1981) — link
- Scalable training of L1 -regularized log-linear models (2007)
- Algorithms on Strings, Trees and Sequences (1997)
- Yara Parser: A Fast and Accurate Dependency Parser (2015) — link
2 of 56 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.