Data Quality Monitoring

# Data Quality Monitoring

## Skill Profile
*(Select at least one profile to enable specific modules)*
- [ ] **DevOps**
- [x] **Backend**
- [ ] **Frontend**
- [ ] **AI-RAG**
- [ ] **Security Critical**

## Overview
Data Quality (DQ) Monitoring is the continuous process of validating data against predefined rules and expectations. In a modern data stack, monitoring must happen at every stage: **Ingestion**, **Transformation**, and **Serving**.

**Core Principle**: "Garbage in, garbage out. If data is wrong, analytics, AI, and decisions will be wrong too."

This skill provides comprehensive guidance on implementing data quality monitoring across the data pipeline, from automated testing to real-time anomaly detection.

## Why This Matters
- **<Benefit>**: <short explanation>
- **<Benefit>**: <short explanation>
- **<Benefit>**: <short explanation>

## Core Concepts & Rules

### 1. Core Principles
- Follow established patterns and conventions
- Maintain consistency across codebase
- Document decisions and trade-offs

### 2. Implementation Guidelines
- Start with the simplest viable solution
- Iterate based on feedback and requirements
- Test thoroughly before deployment


## Inputs / Outputs / Contracts
* **Inputs**:
  - <e.g., env vars, request payload, file paths, schema>
* **Entry Conditions**:
  - <Pre-requisites: e.g., Repo initialized, DB running, specific branch checked out>
* **Outputs**:
  - <e.g., artifacts (PR diff, docs, tests, dashboard JSON)>
* **Artifacts Required (Deliverables)**:
  - <e.g., Code Diff, Unit Tests, Migration Script, API Docs>
* **Acceptance Evidence**:
  - <e.g., Test Report (screenshot/log), Benchmark Result, Security Scan Report>
* **Success Criteria**:
  - <e.g., p95 < 300ms, coverage ≥ 80%>

## Skill Composition
* **Depends on**: None
* **Compatible with**: None
* **Conflicts with**: None
* **Related Skills**: None

## Quick Start
#

## Assumptions
- Data streams are accessible for monitoring
- Historical data is available for baselines
- Team has capacity to respond to alerts
- Monitoring tools are compatible with data stack

## Compatibility
- Works with any SQL database
- Compatible with all major data platforms
- Framework-agnostic approach
- Adaptable to real-time and batch systems

## Test Scenario Matrix
| Scenario | Test Case | Expected Outcome |
|----------|-----------|------------------|
| Null values | Monitor for nulls in required fields | Alert when threshold exceeded |
| Freshness | Monitor data latency | Alert if data is stale |
| Volume | Monitor row count | Alert on sudden drops or spikes |
| Distribution | Monitor statistical properties | Alert on significant shifts |
| Schema | Monitor schema changes | Alert on column additions/removals |
| Accuracy | Compare to source system | Alert on mismatches |

## Technical Guardrails & Security Threat Model

### 1. Security & Privacy (Threat Model)
* **Top Threats**: Injection attacks, authentication bypass, data exposure
- [ ] **Data Handling**: Sanitize all user inputs to prevent Injection attacks. Never log raw PII
- [ ] **Secrets Management**: No hardcoded API keys. Use Env Vars/Secrets Manager
- [ ] **Authorization**: Validate user permissions before state changes

### 2. Performance & Resources
- [ ] **Execution Efficiency**: Consider time complexity for algorithms
- [ ] **Memory Management**: Use streams/pagination for large data
- [ ] **Resource Cleanup**: Close DB connections/file handlers in finally blocks

### 3. Architecture & Scalability
- [ ] **Design Pattern**: Follow SOLID principles, use Dependency Injection
- [ ] **Modularity**: Decouple logic from UI/Frameworks

### 4. Observability & Reliability
- [ ] **Logging Standards**: Structured JSON, include trace IDs `request_id`
- [ ] **Metrics**: Track `error_rate`, `latency`, `queue_depth`
- [ ] **Error Handling**: Standardized error codes, no bare except
- [ ] **Observability Artifacts**:
    - **Log Fields**: timestamp, level, message, request_id
    - **Metrics**: request_count, error_count, response_time
    - **Dashboards/Alerts**: High Error Rate > 5%


## Agent Directives
1. **Always monitor** - Never skip monitoring for production data
2. **Alert on failures** - All quality failures should trigger alerts
3. **Block downstream** - Prevent bad data from propagating
4. **Track metrics** - Monitor quality over time
5. **Iterate on rules** - Update rules based on incidents

## Definition of Done
Data quality monitoring implementation is complete when:

- [ ] Monitoring rules defined for all critical tables
- [ ] Real-time checks implemented where needed
- [ ] Alerting configured for all quality failures
- [ ] Dashboard built and accessible to stakeholders
- [ ] Incident procedures documented
- [ ] Team trained on monitoring tools
- [ ] Regular review schedule established
- [ ] Integration with pipeline orchestration
- [ ] Performance optimization completed
- [ ] Continuous improvement process in place

## Anti-patterns
1. **Monitoring only at load** - Monitor at every stage
2. **Silent failures** - All failures should alert
3. **Over-monitoring** - Too many checks cause alert fatigue
4. **Ignoring alerts** - Every alert needs attention
5. **Static rules** - Rules must evolve with data

## Reference Links
#

## Versioning
This skill follows semantic versioning (MAJOR.MINOR.PATCH):
- **MAJOR**: Breaking changes to procedures or standards
- **MINOR**: New monitoring methods or significant enhancements
- **PATCH**: Bug fixes or documentation updates