Paper: arXiv 2512.05559

Authors: Devender Saini, Bhavika Jain, Nitish Ujjwal, Philip Sommer, Dan Romuald Mbanga, Dhagash Mehta

Abstract

In regulated domains such as finance, the integrity and governance of data pipelines are critical - yet existing systems treat data quality control (QC) as an isolated preprocessing step rather than a first-class system component. We present a unified AI-driven Data QC and DataOps Management framework that embeds rule-based, statistical, and AI-based QC methods into a continuous, governed layer spanning ingestion, model pipelines, and downstream applications. Our architecture integrates open-source tools with custom modules for profiling, audit logging, breach handling, configuration-driven policies, and dynamic remediation. We demonstrate deployment in a production-grade financial setup: handling streaming and tabular data across multiple asset classes and transaction streams, with configurable thresholds, cloud-native storage interfaces, and automated alerts. We show empirical gains in anomaly detection recall, reduction of manual remediation effort, and improved auditability and traceability in high-throughput data workflows. By treating QC as a system concern rather than an afterthought, our framework provides a foundation for trustworthy, scalable, and compliant AI pipelines in regulated environments.

Complexity vs Empirical Score

  • Math Complexity: 1.0/10
  • Empirical Rigor: 8.5/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper focuses on system architecture and operational implementation with minimal advanced mathematics, emphasizing practical deployment, data handling, and empirical performance gains in a production environment.

Research Flowchart

  flowchart TD
  Start["Research Goal:<br>Unified AI Framework for Data QC & DataOps in Regulated Environments"] --> Input["Data Inputs:<br>Streaming & Tabular Data<br>(Multi-Asset)"]
  
  Input --> Method["Methodology:<br>AI-Driven QC Layer<br>(Rule-Based + Statistical + AI Methods)"]
  
  Method --> Proc["Computational Processes:<br>1. Profiling & Audit Logging<br>2. Breach Handling & Dynamic Remediation<br>3. Configurable Policy Enforcement"]
  
  Proc --> Outcome["Key Outcomes:<br>• Higher Anomaly Detection Recall<br>• Reduced Manual Remediation<br>• Improved Auditability & Traceability"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes a unified AI-driven Data Quality Control and DataOps Management framework that integrates rule‑based, statistical, and AI‑based QC methods into a continuously governed layer across ingestion, model pipelines, and downstream applications.

Source sentence
We present a unified AI-driven Data QC and DataOps Management framework that embeds rule-based, statistical, and AI-based QC methods into a continuous, governed layer spanning ingestion, model pipelines, and downstream applications.

Assumes 3

  • Missing data in financial pipelines is rarely MCAR; it is often MAR or MNAR due to observable or unobserved market behavior. data
    Source sentence
    In financial data pipelines, missingness is rarely completely at random (MCAR); it is often missing at random (MAR) due to observable factors, or missing not at random (MNAR) due to unobserved market behavior or reporting practices.
  • Rule‑based and statistical QC checks rely on predefined thresholds and historical bounds computed over prior observations. model
    Source sentence
    do_outlier_detection_std to flag outliers exceeding mean-based thresholds derived from standard deviation; do_outlier_detection_range to detect values lying outside historical minimum and maximum bounds computed over ten prior observations per feature
  • Imputation‑aware QC treats imputation as a governed process rather than a one‑time correction. model
    Source sentence
    the proposed QC framework includes an imputation-aware, model-based QC layer designed to treat imputation as a governed process rather than a one-time correction.

Data

  • Corporate Bonds Indices · Kaggle benchmark public
    Source sentence
    on the “Corporate Bonds Indices” dataset on Kaggle (DenzilG, 2023).
  • Fraud Dataset · Jiang et al., 2022 benchmark public
    Source sentence
    More specifically, the Fraud dataset contains about 284807 samples with 29 features and only 0.17% (492) anomalies

Costs

Not applicable The paper reports no trading or execution result.

Finds 2

  • Imputation‑aware QC raised precision and recall from 0.74/0.53 to about 0.90/0.90 and cut false‑positive rate from ~48% to ~10%, a ~5× reduction in false alerts and >130% F1 improvement. performance
    Source sentence
    Introducing imputation-aware pre-processing significantly improved both precision and recall to approximately 0.9, while reducing the false-positive rate to below 10%. This corresponds to a ∼5× reduction in false alerts and a relative F1-score improvement of more than 130%.
  • The unified framework demonstrated empirical gains in anomaly detection recall, reduction of manual remediation effort, and improved auditability in high‑throughput data workflows. other
    Source sentence
    We show empirical gains in anomaly detection recall, reduction of manual remediation effort, and improved auditability and traceability in high-throughput data workflows.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 10 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (44)

  1. Explainable Unsupervised Anomaly Detection with Random Forest (2025) — link
  2. A Survey on Universal Approximation Theorems (2024) — link
  3. Detecting drifts in data streams using Kullback-Leibler (KL) divergence measure for data engineering applications (2024) — link
  4. AI-Powered Data Governance: A Cutting-Edge Method for Ensuring Data Quality for Machine Learning Applications (2024) — link
  5. Data-centric Artificial Intelligence: A Survey (2023) — link
  6. Extrapolation and AI transparency: Why machine learning models should reveal when they make decisions beyond their training (2023) — link
  7. Detecting Anomalies in Financial Data Using Machine Learning Algorithms (2022) — link
  8. Data collection and quality challenges in deep learning: a data-centric AI perspective (2021) — link
and 36 more
  1. Financial Fraud: A Review of Anomaly Detection Techniques and Recent Advances (2021) — link
  2. A survey on missing data in machine learning (2021) — link
  3. “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021) — link
  4. Competing in the Age of AI: Strategy and Leadership When Algorithms and Networks Run the World (2021) — link
  5. Unsupervised anomaly detection for internal auditing: Literature review and research agenda (2021) — link
  6. Data quality problems troubling business and financial researchers: A literature review and synthetic analysis (2020) — link
  7. Overview and Importance of Data Quality for Machine Learning Tasks (2020) — link
  8. Current landscape and influence of big data on finance (2020) — link
  9. Detecting Outliers in High Dimensional Data Sets using Z-Score Methodology (2019) — link
  10. Data Quality: Dimensions, Measurement, Strategy, Management and Governance (Book Review) (2019) — link
  11. PyOD: A Python Toolbox for Scalable Outlier Detection (2019) — link
  12. Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models (2018) — link
  13. XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning (2018) — link
  14. A survey of anomaly detection techniques in financial domain (2016)
  15. Deep Reinforcement Learning with Double Q-Learning (2015) — link
  16. A Modified Approach for Detection of Outliers (2015) — link
  17. Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares (2014) — link
  18. Outlier Analysis (2013) — link
  19. MICE: Multivariate Imputation by Chained Equations in R (2011) — link
  20. MissForest - non-parametric missing value imputation for mixed-type data (2011) — link
  21. Scikit-learn: Machine Learning in Python (2011) — link
  22. Types of Machine Learning Algorithms (2010) — link
  23. Principal Component Analysis (2003) — link
  24. Statistical Methods for Quality Improvement (2001) — link
  25. Approximation capabilities of multilayer feedforward networks (1991) — link
  26. Machine Learning for Anomaly Detection: A Systematic Review (2021) — link
  27. Validating Data and Models in Continuous ML Pipelines (2021)
  28. Ethical issues in data science: From privacy to bias (2020)
  29. DataOps - Towards a Definition (2018)
  30. Great Expectations (2015) — link
  31. Gener-ative adversarial networks (2014)
  32. FederalReserveSystem (2011)
  33. Much ado about nothing: A comparison of missing data methods and software to fit incomplete data regression models (2007)
  34. Approximation by superpositions of a sig-moidal function (1989)
  35. AI Factory: Theories, Applications and Case Studies
  36. Enhancing financial security: Data science’s role in risk management and fraud detection

0 of 44 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.