Paper: arXiv 2311.08533

Authors: Ixandra Achitouv, Dragos Gorduza, Antoine Jacquier

Abstract

This article provides an understanding of Natural Language Processing techniques in the framework of financial regulation, more specifically in order to perform semantic matching search between rules and policy when no dataset is available for supervised learning. We outline how to outperform simple pre-trained sentences-transformer models using freely available resources and explain the mathematical concepts behind the key building blocks of Natural Language Processing.

Complexity vs Empirical Score

  • Math Complexity: 6.0/10
  • Empirical Rigor: 3.0/10
  • Quadrant: Lab Rats — high math complexity, limited empirical testing

Why this score: The paper explains advanced NLP concepts and mathematical formulations like co-occurrence matrices, SVD, and Skip-gram training objectives, but provides no implementation details, backtesting results, or code, focusing instead on theoretical understanding for financial regulation.

Research Flowchart

  flowchart TD
  Start["Research Goal: Semantic Matching<br>for Financial Regulation<br>(No Labeled Dataset)"]
  
  Data -->|"Free Text Resources"| Embedding
  subgraph Data ["Data / Inputs"]
      Rules["Regulatory Rules"]
      Policies["Policy Documents"]
  end

  subgraph Methodology ["Methodology"]
      Embedding["Fine-tune Sentence Transformers<br>(Unsupervised Learning)"]
      Match["Semantic Matching<br>Vector Similarity Search"]
      Eval["Evaluate against<br>Baseline Models"]
  end

  subgraph Outcomes ["Key Findings / Outcomes"]
      Result["Outperforms<br>Pre-trained Baselines"]
      Insight["Mathematical Understanding<br>of NLP Building Blocks"]
  end

  Start --> Data
  Embedding --> Match
  Match --> Eval
  Eval --> Result
  Eval --> Insight

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes an adaptive pre‑training and a fully unsupervised domain‑adaptation method (GPL) to improve semantic matching between financial rules and policies using freely available resources.

Source sentence
we explore the results of performing an Adaptative Pre-Training method and a purely unsupervised approach, using GPL.

Assumes 2

  • A pseudo‑Markovian (k‑gram) approximation is assumed for estimating word probabilities. model
    Source sentence
    one usually assumes a pseudo-Markovian approximation (the k-gram model)
  • Pairs are kept only if cosine similarity is above 0.7, an empirical threshold for matching. data
    Source sentence
    For each model, we keep the rule/policy match sentences if the cosine similarity (6) is above 0.7. This is an empirical choice that provides fair matches.

Data

  • FCA Rulebook 2022 · Financial Conduct Authority text-news public
    Source sentence
    The dataset of rules is publicly available from Financial Conduct Authority (FCA) Rulebook 2022. After cleaning and splitting the rules into sentences of length 200, we obtain about 50,000 sentences, composed of 21,914 rule IDs.
  • FinregE policies · FinregE text-news commercial
    Source sentence
    The policies’ dataset was obtained from FinregE and is composed of 2,374 policies that may not cover all of the newly edited FCA

Costs

Not applicable The paper reports no trading or execution result.

Finds 1

  • All‑MiniLM‑L6‑v2 with Step 1 & 2 improves Score 1 from 0.21 to 0.27 (29% increase) and Score 2 from 0.46 to 0.56 (22% increase). performance
    Source sentence
    all-MiniLM-L6-v2 + Step 1 & 2 0.27 0.56 Improvement 29% 22%

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 4 related papers). Coverage is partial, so this is neither replication nor refutation.

Cited by 1 paper in the archive (1 by independent authors)

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (24)

  1. An improved model of semantic similarity based on lexical co-occurrence (2006)
  2. SciBERT: A Pretrained Language Model for Scientific Text (2019)
  3. Language models are few-shot learners (2020)
  4. Indexing by latent semantic analysis (1990)
  5. Producing high-dimensional semantic spaces from lexical co-occurrence (1996)
  6. A maximum entropy approach to adaptive statistical language modelling (1996)
  7. A closer look at skip-gram modelling. (2006)
  8. Efficient estimation of word representations in vector space (2013)
and 16 more
  1. Distributed representations of words and phrases and their compositionality (2013)
  2. Formal algorithms for transformers (2022)
  3. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring (2019)
  4. Bert: Pre-training of deep bidirectional transformers for language understanding (2018)
  5. Glove: Global vectors for word representation (2014)
  6. Sentence- Bert : Sentence embeddings using S iamese Bert -networks (2019)
  7. Attention is all you need (2017)
  8. BioBert : a pre-trained biomedical language representation model for biomedical text mining (2020)
  9. Tsdae: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning (2021)
  10. Facenet: A unified embedding for face recognition and clustering (2015)
  11. Efficient natural language response suggestion for smart reply (2017)
  12. GPL : Generative pseudo labeling for unsupervised domain adaptation of dense retrieval (2022)
  13. Exploring the limits of transfer learning with a unified text-to-text transformer (2020)
  14. Molecule attention transformer (2002)
  15. An image is worth 16$\times$16 words: Transformers for image recognition at scale (2010)
  16. Studies in linguistic analysis (1957)

0 of 24 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.