Paper: arXiv 2603.10807

Authors: Fabrizio Dimino, Bhaskarjit Sarmah, Stefano Pasquali

Abstract

The rapid adoption of large language models (LLMs) in financial services introduces new operational, regulatory, and security risks. Yet most red-teaming benchmarks remain domain-agnostic and fail to capture failure modes specific to regulated BFSI settings, where harmful behavior can be elicited through legally or professionally plausible framing. We propose a risk-aware evaluation framework for LLM security failures in Banking, Financial Services, and Insurance (BFSI), combining a domain-specific taxonomy of financial harms, an automated multi-round red-teaming pipeline, and an ensemble-based judging protocol. We introduce the Risk-Adjusted Harm Score (RAHS), a risk-sensitive metric that goes beyond success rates by quantifying the operational severity of disclosures, accounting for mitigation signals, and leveraging inter-judge agreement. Across diverse models, we find that higher decoding stochasticity and sustained adaptive interaction not only increase jailbreak success, but also drive systematic escalation toward more severe and operationally actionable financial disclosures. These results expose limitations of single-turn, domain-agnostic security evaluation and motivate risk-sensitive assessment under prolonged adversarial pressure for real-world BFSI deployment.

Complexity vs Empirical Score

  • Math Complexity: 3.5/10
  • Empirical Rigor: 7.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper provides a highly practical and novel evaluation framework for financial AI safety, introducing a domain-specific taxonomy (FinRedTeamBench) and a novel Risk-Adjusted Harm Score (RAHS). While mathematically light, relying primarily on heuristic scoring formulas and ensemble statistics, the empirical testing is thorough, relevant, and addresses critical regulatory gaps in LLM red-teaming.

Research Flowchart

  flowchart TD
    A[Research Goal: Evaluate LLM Security Risks in Regulated BFSI Settings] --> B[Data Inputs: BFSI Financial Harm Taxonomy & Domain Attack Prompts]
    B --> C[Method: Automated Multi-Round Adaptive Red-Teaming Pipeline]
    C --> D[Process: Ensemble-Based Judging Protocol & Severity Assessment]
    D --> E[Metric Calculation: Risk-Adjusted Harm Score - RAHS]
    E --> F[Key Outcome: High Stochasticity & Multi-Turn Attack Escalates Disclosure Severity]

Cited References (38)

  1. The Red-Teaming Resistance Leaderboard (2024)
  2. The MLCommons AI Safety Benchmark (2024)
  3. Stealthy jailbreak attacks on large language models via benign data mirroring (2025)
  4. Large Language Models in Finance (FinLLMs) (2025)
  5. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models (2023)
  6. Safetybench: Evaluating the safety of large language models (2024)
  7. Llm360: Towards fully transparent open-source llms (2023) — link
  8. Red teaming language models with language models (2022)
and 30 more
  1. Jailbreaking black box large language models in twenty queries (2025)
  2. Automated red teaming with goat: the generative offensive agent tester (2024) — link
  3. RedAgent: An Autonomous Agent for Context-aware Red Teaming of LLM Jailbreaks (2026)
  4. X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents (2025)
  5. Easyjailbreak: A unified framework for jailbreaking large language models (2024) — link
  6. Artificial intelligence in financial services (2026)
  7. Great, now write an article about that: The crescendo $\ $Multi-Turn$\ $$\ $LLM$\ $ jailbreak attack (2025)
  8. When llm meets drl: Advancing jailbreaking efficiency via drl-guided search (2024)
  9. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal (2024) — link
  10. Universal and transferable adversarial attacks on aligned language models (2023) — link
  11. Uncovering the vulnerability of large language models in the financial domain via risk concealment (2025) — link
  12. Ignore previous prompt: Attack techniques for language models (2022) — link
  13. Jailbroken: How does llm safety training fail? (2023)
  14. Leakagent: Rl-based red-teaming agent for llm privacy leakage (2024) — link
  15. CNFinBench: A Benchmark for Safety and Compliance of Large Language Models in Finance (2025) — link
  16. Agentvigil: Generic black-box red-teaming for indirect prompt injection against llm agents (2025) — link
  17. Autodan: Generating stealthy jailbreak prompts on aligned large language models (2024)
  18. Jailbreakbench: An open robustness benchmark for jailbreaking large language models (2024)
  19. RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models (2026) — link
  20. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)
  21. Large language models in finance: A survey (2023)
  22. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices (2024)
  23. Red-Teaming Financial AI Agents: Stress-Testing Governance Protections in LLMs Against Market Manipulation and Regulatory Evasion (2026)
  24. SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types (2024)
  25. OWASP Top 10 for LLM Applications 2025 (2025)
  26. Attackeval: How to evaluate the effectiveness of jailbreak attacking on large language models (2025)
  27. Jades: A universal framework for jailbreak assessment via decompositional scoring (2025) — link
  28. Bells: A framework towards future proof benchmarks for the evaluation of llm safeguards (2024) — link
  29. h4rm3l: A language for composable jailbreak attack synthesis (2025)
  30. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models (2024)

0 of 38 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2518 linked papers.