Paper: arXiv 2311.08533
Authors: Ixandra Achitouv, Dragos Gorduza, Antoine Jacquier
Abstract
This article provides an understanding of Natural Language Processing techniques in the framework of financial regulation, more specifically in order to perform semantic matching search between rules and policy when no dataset is available for supervised learning. We outline how to outperform simple pre-trained sentences-transformer models using freely available resources and explain the mathematical concepts behind the key building blocks of Natural Language Processing.
Complexity vs Empirical Score
- Math Complexity: 6.0/10
- Empirical Rigor: 3.0/10
- Quadrant: Lab Rats — high math complexity, limited empirical testing
Why this score: The paper explains advanced NLP concepts and mathematical formulations like co-occurrence matrices, SVD, and Skip-gram training objectives, but provides no implementation details, backtesting results, or code, focusing instead on theoretical understanding for financial regulation.
Research Flowchart
flowchart TD
Start["Research Goal: Semantic Matching<br>for Financial Regulation<br>(No Labeled Dataset)"]
Data -->|"Free Text Resources"| Embedding
subgraph Data ["Data / Inputs"]
Rules["Regulatory Rules"]
Policies["Policy Documents"]
end
subgraph Methodology ["Methodology"]
Embedding["Fine-tune Sentence Transformers<br>(Unsupervised Learning)"]
Match["Semantic Matching<br>Vector Similarity Search"]
Eval["Evaluate against<br>Baseline Models"]
end
subgraph Outcomes ["Key Findings / Outcomes"]
Result["Outperforms<br>Pre-trained Baselines"]
Insight["Mathematical Understanding<br>of NLP Building Blocks"]
end
Start --> Data
Embedding --> Match
Match --> Eval
Eval --> Result
Eval --> Insight
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes an adaptive pre‑training and a fully unsupervised domain‑adaptation method (GPL) to improve semantic matching between financial rules and policies using freely available resources.
Source sentence
we explore the results of performing an Adaptative Pre-Training method and a purely unsupervised approach, using GPL.
Assumes 2
- A pseudo‑Markovian (k‑gram) approximation is assumed for estimating word probabilities. model
Source sentence
one usually assumes a pseudo-Markovian approximation (the k-gram model)
- Pairs are kept only if cosine similarity is above 0.7, an empirical threshold for matching. data
Source sentence
For each model, we keep the rule/policy match sentences if the cosine similarity (6) is above 0.7. This is an empirical choice that provides fair matches.
Data
- FCA Rulebook 2022 text-news public
Source sentence
The dataset of rules is publicly available from Financial Conduct Authority (FCA) Rulebook 2022. After cleaning and splitting the rules into sentences of length 200, we obtain about 50,000 sentences, composed of 21,914 rule IDs.
- FinregE policies text-news commercial
Source sentence
The policies’ dataset was obtained from FinregE and is composed of 2,374 policies that may not cover all of the newly edited FCA
Costs
Not applicable
Finds 1
- All‑MiniLM‑L6‑v2 with Step 1 & 2 improves Score 1 from 0.21 to 0.27 (29% increase) and Score 2 from 0.46 to 0.56 (22% increase). performance
Source sentence
all-MiniLM-L6-v2 + Step 1 & 2 0.27 0.56 Improvement 29% 22%
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 4 related papers). Coverage is partial, so this is neither replication nor refutation.
1 paper in the archive
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (24)
- An improved model of semantic similarity based on lexical co-occurrence (2006)
- SciBERT: A Pretrained Language Model for Scientific Text (2019)
- Language models are few-shot learners (2020)
- Indexing by latent semantic analysis (1990)
- Producing high-dimensional semantic spaces from lexical co-occurrence (1996)
- A maximum entropy approach to adaptive statistical language modelling (1996)
- A closer look at skip-gram modelling. (2006)
- Efficient estimation of word representations in vector space (2013)
and 16 more
- Distributed representations of words and phrases and their compositionality (2013)
- Formal algorithms for transformers (2022)
- Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring (2019)
- Bert: Pre-training of deep bidirectional transformers for language understanding (2018)
- Glove: Global vectors for word representation (2014)
- Sentence- Bert : Sentence embeddings using S iamese Bert -networks (2019)
- Attention is all you need (2017)
- BioBert : a pre-trained biomedical language representation model for biomedical text mining (2020)
- Tsdae: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning (2021)
- Facenet: A unified embedding for face recognition and clustering (2015)
- Efficient natural language response suggestion for smart reply (2017)
- GPL : Generative pseudo labeling for unsupervised domain adaptation of dense retrieval (2022)
- Exploring the limits of transfer learning with a unified text-to-text transformer (2020)
- Molecule attention transformer (2002)
- An image is worth 16$\times$16 words: Transformers for image recognition at scale (2010)
- Studies in linguistic analysis (1957)
0 of 24 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.