Paper: arXiv 2512.05559
Authors: Devender Saini, Bhavika Jain, Nitish Ujjwal, Philip Sommer, Dan Romuald Mbanga, Dhagash Mehta
Abstract
In regulated domains such as finance, the integrity and governance of data pipelines are critical - yet existing systems treat data quality control (QC) as an isolated preprocessing step rather than a first-class system component. We present a unified AI-driven Data QC and DataOps Management framework that embeds rule-based, statistical, and AI-based QC methods into a continuous, governed layer spanning ingestion, model pipelines, and downstream applications. Our architecture integrates open-source tools with custom modules for profiling, audit logging, breach handling, configuration-driven policies, and dynamic remediation. We demonstrate deployment in a production-grade financial setup: handling streaming and tabular data across multiple asset classes and transaction streams, with configurable thresholds, cloud-native storage interfaces, and automated alerts. We show empirical gains in anomaly detection recall, reduction of manual remediation effort, and improved auditability and traceability in high-throughput data workflows. By treating QC as a system concern rather than an afterthought, our framework provides a foundation for trustworthy, scalable, and compliant AI pipelines in regulated environments.
Complexity vs Empirical Score
- Math Complexity: 1.0/10
- Empirical Rigor: 8.5/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper focuses on system architecture and operational implementation with minimal advanced mathematics, emphasizing practical deployment, data handling, and empirical performance gains in a production environment.
Research Flowchart
flowchart TD Start["Research Goal:<br>Unified AI Framework for Data QC & DataOps in Regulated Environments"] --> Input["Data Inputs:<br>Streaming & Tabular Data<br>(Multi-Asset)"] Input --> Method["Methodology:<br>AI-Driven QC Layer<br>(Rule-Based + Statistical + AI Methods)"] Method --> Proc["Computational Processes:<br>1. Profiling & Audit Logging<br>2. Breach Handling & Dynamic Remediation<br>3. Configurable Policy Enforcement"] Proc --> Outcome["Key Outcomes:<br>• Higher Anomaly Detection Recall<br>• Reduced Manual Remediation<br>• Improved Auditability & Traceability"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes a unified AI-driven Data Quality Control and DataOps Management framework that integrates rule‑based, statistical, and AI‑based QC methods into a continuously governed layer across ingestion, model pipelines, and downstream applications.
Source sentence
We present a unified AI-driven Data QC and DataOps Management framework that embeds rule-based, statistical, and AI-based QC methods into a continuous, governed layer spanning ingestion, model pipelines, and downstream applications.
Assumes 3
- Missing data in financial pipelines is rarely MCAR; it is often MAR or MNAR due to observable or unobserved market behavior. data
Source sentence
In financial data pipelines, missingness is rarely completely at random (MCAR); it is often missing at random (MAR) due to observable factors, or missing not at random (MNAR) due to unobserved market behavior or reporting practices.
- Rule‑based and statistical QC checks rely on predefined thresholds and historical bounds computed over prior observations. model
Source sentence
do_outlier_detection_std to flag outliers exceeding mean-based thresholds derived from standard deviation; do_outlier_detection_range to detect values lying outside historical minimum and maximum bounds computed over ten prior observations per feature
- Imputation‑aware QC treats imputation as a governed process rather than a one‑time correction. model
Source sentence
the proposed QC framework includes an imputation-aware, model-based QC layer designed to treat imputation as a governed process rather than a one-time correction.
Data
- Corporate Bonds Indices benchmark public
Source sentence
on the “Corporate Bonds Indices” dataset on Kaggle (DenzilG, 2023).
- Fraud Dataset benchmark public
Source sentence
More specifically, the Fraud dataset contains about 284807 samples with 29 features and only 0.17% (492) anomalies
Costs
Not applicable
Finds 2
- Imputation‑aware QC raised precision and recall from 0.74/0.53 to about 0.90/0.90 and cut false‑positive rate from ~48% to ~10%, a ~5× reduction in false alerts and >130% F1 improvement. performance
Source sentence
Introducing imputation-aware pre-processing significantly improved both precision and recall to approximately 0.9, while reducing the false-positive rate to below 10%. This corresponds to a ∼5× reduction in false alerts and a relative F1-score improvement of more than 130%.
- The unified framework demonstrated empirical gains in anomaly detection recall, reduction of manual remediation effort, and improved auditability in high‑throughput data workflows. other
Source sentence
We show empirical gains in anomaly detection recall, reduction of manual remediation effort, and improved auditability and traceability in high-throughput data workflows.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 10 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (44)
- Explainable Unsupervised Anomaly Detection with Random Forest (2025) — link
- A Survey on Universal Approximation Theorems (2024) — link
- Detecting drifts in data streams using Kullback-Leibler (KL) divergence measure for data engineering applications (2024) — link
- AI-Powered Data Governance: A Cutting-Edge Method for Ensuring Data Quality for Machine Learning Applications (2024) — link
- Data-centric Artificial Intelligence: A Survey (2023) — link
- Extrapolation and AI transparency: Why machine learning models should reveal when they make decisions beyond their training (2023) — link
- Detecting Anomalies in Financial Data Using Machine Learning Algorithms (2022) — link
- Data collection and quality challenges in deep learning: a data-centric AI perspective (2021) — link
and 36 more
- Financial Fraud: A Review of Anomaly Detection Techniques and Recent Advances (2021) — link
- A survey on missing data in machine learning (2021) — link
- “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI (2021) — link
- Competing in the Age of AI: Strategy and Leadership When Algorithms and Networks Run the World (2021) — link
- Unsupervised anomaly detection for internal auditing: Literature review and research agenda (2021) — link
- Data quality problems troubling business and financial researchers: A literature review and synthetic analysis (2020) — link
- Overview and Importance of Data Quality for Machine Learning Tasks (2020) — link
- Current landscape and influence of big data on finance (2020) — link
- Detecting Outliers in High Dimensional Data Sets using Z-Score Methodology (2019) — link
- Data Quality: Dimensions, Measurement, Strategy, Management and Governance (Book Review) (2019) — link
- PyOD: A Python Toolbox for Scalable Outlier Detection (2019) — link
- Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models (2018) — link
- XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning (2018) — link
- A survey of anomaly detection techniques in financial domain (2016)
- Deep Reinforcement Learning with Double Q-Learning (2015) — link
- A Modified Approach for Detection of Outliers (2015) — link
- Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares (2014) — link
- Outlier Analysis (2013) — link
- MICE: Multivariate Imputation by Chained Equations in R (2011) — link
- MissForest - non-parametric missing value imputation for mixed-type data (2011) — link
- Scikit-learn: Machine Learning in Python (2011) — link
- Types of Machine Learning Algorithms (2010) — link
- Principal Component Analysis (2003) — link
- Statistical Methods for Quality Improvement (2001) — link
- Approximation capabilities of multilayer feedforward networks (1991) — link
- Machine Learning for Anomaly Detection: A Systematic Review (2021) — link
- Validating Data and Models in Continuous ML Pipelines (2021)
- Ethical issues in data science: From privacy to bias (2020)
- DataOps - Towards a Definition (2018)
- Great Expectations (2015) — link
- Gener-ative adversarial networks (2014)
- FederalReserveSystem (2011)
- Much ado about nothing: A comparison of missing data methods and software to fit incomplete data regression models (2007)
- Approximation by superpositions of a sig-moidal function (1989)
- AI Factory: Theories, Applications and Case Studies
- Enhancing financial security: Data science’s role in risk management and fraud detection
0 of 44 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.