Paper: arXiv 2512.05559

Authors: Devender Saini, Bhavika Jain, Nitish Ujjwal, Philip Sommer, Dan Romuald Mbanga, Dhagash Mehta

Abstract

In regulated domains such as finance, the integrity and governance of data pipelines are critical - yet existing systems treat data quality control (QC) as an isolated preprocessing step rather than a first-class system component. We present a unified AI-driven Data QC and DataOps Management framework that embeds rule-based, statistical, and AI-based QC methods into a continuous, governed layer spanning ingestion, model pipelines, and downstream applications. Our architecture integrates open-source tools with custom modules for profiling, audit logging, breach handling, configuration-driven policies, and dynamic remediation. We demonstrate deployment in a production-grade financial setup: handling streaming and tabular data across multiple asset classes and transaction streams, with configurable thresholds, cloud-native storage interfaces, and automated alerts. We show empirical gains in anomaly detection recall, reduction of manual remediation effort, and improved auditability and traceability in high-throughput data workflows. By treating QC as a system concern rather than an afterthought, our framework provides a foundation for trustworthy, scalable, and compliant AI pipelines in regulated environments.

Complexity vs Empirical Score

  • Math Complexity: 1.0/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — practical and empirical, lighter on theory

Why this score: The paper focuses on system architecture and operational implementation with minimal advanced mathematics, emphasizing practical deployment, data handling, and empirical performance gains in a production environment.

Research Flowchart

  flowchart TD
  Start[""Research Goal:
  Unified AI Framework for Data QC & DataOps in Regulated Environments""] --> Input[""Data Inputs:
  Streaming & Tabular Data
  (Multi-Asset)""]
  
  Input --> Method[""Methodology:
  AI-Driven QC Layer
  (Rule-Based + Statistical + AI Methods)""]
  
  Method --> Proc[""Computational Processes:
  1. Profiling & Audit Logging
  2. Breach Handling & Dynamic Remediation
  3. Configurable Policy Enforcement""]
  
  Proc --> Outcome[""Key Outcomes:
  • Higher Anomaly Detection Recall
  • Reduced Manual Remediation
  • Improved Auditability & Traceability""]