Paper: arXiv 2502.07131
Authors: Yewon Hwang, Sungbum Jung, Hanwool Lee, Sara Yu
Abstract
Domain specificity of embedding models is critical for effective performance. However, existing benchmarks, such as FinMTEB, are primarily designed for high-resource languages, leaving low-resource settings, such as Korean, under-explored. Directly translating established English benchmarks often fails to capture the linguistic and cultural nuances present in low-resource domains. In this paper, titled TWICE: What Advantages Can Low-Resource Domain-Specific Embedding Models Bring? A Case Study on Korea Financial Texts, we introduce KorFinMTEB, a novel benchmark for the Korean financial domain, specifically tailored to reflect its unique cultural characteristics in low-resource languages. Our experimental results reveal that while the models perform robustly on a translated version of FinMTEB, their performance on KorFinMTEB uncovers subtle yet critical discrepancies, especially in tasks requiring deeper semantic understanding, that underscore the limitations of direct translation. This discrepancy highlights the necessity of benchmarks that incorporate language-specific idiosyncrasies and cultural nuances. The insights from our study advocate for the development of domain-specific evaluation frameworks that can more accurately assess and drive the progress of embedding models in low-resource settings.
Complexity vs Empirical Score
- Math Complexity: 3.5/10
- Empirical Rigor: 6.0/10 (capped from 7.2: no out-of-sample test; see penalty flags)
- Quadrant: Street Traders — empirical and applied, lower math complexity (effective empirical rigor capped from 7.2)
Why this score: The paper is heavy on empirical evaluation, featuring a novel benchmark (KorFinMTEB), multiple datasets, and comparative results on SOTA models, but involves almost no advanced mathematics, focusing instead on NLP engineering and data curation.
Research Flowchart
flowchart TD
A["Research Goal: Evaluate advantages of domain-specific embedding models for low-resource Korean financial text"] --> B["KorFinMTEB Benchmark Creation<br>(Cultural & Linguistic Nuances)"]
A --> C["Translated FinMTEB Benchmark<br>(Baseline Comparison)"]
B --> D["Evaluation & Analysis<br>(Korean Financial Embeddings)"]
C --> D
D --> E{"Key Findings: Discrepancy & Necessity"}
E --> F["Models perform robustly on Translated FinMTEB<br>(Superficial Understanding)"]
E --> G["Models struggle on KorFinMTEB<br>(Reveals Semantic & Cultural Gaps)"]
G --> H["Advocates for Domain-Specific Benchmarks<br>(Accurate Low-Resource Assessment)"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes KorFinMTEB, a novel benchmark for Korean financial texts that captures language-specific and cultural nuances.
Source sentence
we introduce KorFinMTEB, a novel benchmark for the Korean financial domain, specifically tailored to reflect its unique cultural characteristics in low-resource languages.
Assumes 2
- Direct translation of existing benchmarks fails to capture cultural nuances in low‑resource financial texts. data
Source sentence
Directly translating existing benchmarks often introduces context loss and fails to capture cultural nuances (Son et al., 2024a;b)—a critical issue in financial texts where precise terminology is paramount.
- Performance differences between translated and native benchmarks will reveal benefits of authentic domain‑specific data. model
Source sentence
Our primary hypothesis is that performance differences will emerge between the two datasets, highlighting the benefits of using authentic, domain-specific data over simple synthetic translations.
Data
- KorFinMTEB benchmark public
Source sentence
The dataset of KorFinMTEB benchmark are fully open-sourced and publicly available, ensuring reproducibility and transparency.
Costs
Not applicable
Finds 4
- Models show a 5–8% performance drop on deeper semantic tasks when evaluated on KorFinMTEB versus the translated benchmark. performance
Source sentence
models consistently exhibited a 5–8% performance drop on KorFinMTEB.
- Retrieval tasks are more challenging on the translated benchmark due to translation artifacts. performance
Source sentence
retrieval tasks (e.g., TATQA-Retrieval-ko and FinNews-Retrieval-ko) proved more challenging on Translated-FinMTEB.
- Korean‑fine‑tuned model kure‑v1 demonstrates more robust performance on KorFinMTEB. performance
Source sentence
models fine-tuned on Korean data (e.g., kure-v1) demonstrated more robust performance, emphasizing the benefits of language-specific training for domain-specific applications.
- KorFinMTEB more faithfully reflects the complexities of Korean financial discourse, providing a more reliable evaluation framework. performance
Source sentence
our results provide compelling evidence that KorFinMTEB more faithfully reflects the complexities of Korean financial discourse and serves as a more reliable evaluation framework for embedding models in low-resource settings.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 29 related papers). Coverage is partial, so this is neither replication nor refutation.
1 paper in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (23)
- Do We Need Domain-Specific Embedding Models? An Empirical Investigation (2024) — link
- MTEB: Massive Text Embedding Benchmark (2023) — link
- KMMLU: Measuring Massive Multitask Language Understanding in Korean (2024) — link
- HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models (2024) — link
- Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives (2024) — link
- Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency (2024)
- BloombergGPT: A Large Language Model for Finance (2023) — link
- When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages (2023) — link
and 15 more
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019) — link
- SimCSE: Simple Contrastive Learning of Sentence Embeddings (2022) — link
- FinBERT: Financial Sentiment Analysis with Pre-trained Language Models (2019) — link
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining (2019) — link
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024) — link
- SGPT: GPT Sentence Embeddings for Semantic Search (2022) — link
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2021) — link
- Efficient Estimation of Word Representations in Vector Space (2013) — link
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019) — link
- Removing Non-Stationary Knowledge From Pre-Trained Language Models for Entity-Level Sentiment Classification in Finance (2023) — link
- Making Text Embedders Few-Shot Learners (2024) — link
- Towards general text embeddings with multi-stage contrastive learning (2023) — link
- Improving Text Embeddings with Large Language Models (2023) — link
- One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2023) — link
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020) — link
0 of 23 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.