Data Quality Policy

Data Quality policies validate your data against predefined quality rules to ensure accuracy, completeness, and consistency. These policies check individual data points to identify valid and invalid records based on criteria you define.

• Policy: A set of Rules and Rule Sets that define quality expectations for a dataset.

• Rule: A specific condition or check that data must meet to be considered valid (such as null checks, schema consistency, value ranges, uniqueness, or pattern matching).

Common Use Cases

• Verify customer data completeness before sending records to downstream systems.

• Validate that transaction amounts fall within expected numerical ranges.

• Check that email addresses and phone numbers adhere to standard formatting.

• Flag unexpected null values in primary keys or identifiers.

• Enforce business logic constraints (such as preventing future order dates).

Create a Data Quality policy

You can initiate policy creation either from the Policies menu or directly from an asset's detail page.

1. Start policy creation

• From Manage Policies:

  1. In the left navigation menu, go to Data Observability Cloud > Governance > Policies.

  2. Select Add Policy in the top-right corner.

  3. Select Data Quality as the policy type.

  4. Choose the target datasets (assets) to monitor.

• From Asset Details:

  • In the Overview tab, select Actions > Add Data Quality Policy.

  • Alternatively, go to the Policies tab and select Add Policy > Data Quality.

2. Configure data selection

Specify which records or columns ADOC evaluates:

• Column-based selection: Select specific columns directly. Best for simple field validations.

• SQL-based selection: Enter a custom SQL query to isolate a subset.

  • Use {{{lower_bound}}} and {{{upper_bound}}} placeholders for incremental runs.

  • Select Validate to verify syntax before continuing.

If you do not use Native SQL, ensure your query syntax matches your underlying data platform (such as Snowflake or BigQuery).

3. Select an execution engine

Choose the compute engine for running policy jobs:

• Spark (default): Executes on ADOC's Spark engine. Recommended for complex transformations, custom resource allocations, and persisting valid/invalid record samples.

• Pushdown: Runs queries directly inside the source database. Recommended for fast checks on large datasets with minimal data movement.

4. Configure quality rules

Add rules manually or apply a Data Policy Template to apply predefined sets of checks.

5. Define evaluation thresholds and scoring

Set policy-level and rule-level pass criteria:

• Success threshold: The minimum percentage score (0–100%) required for a pass.

• Warning threshold (optional): A secondary boundary that flags degrading quality before a complete failure occurs.

6. Configure alerts and notifications (optional)

1. Select an alert severity level: Critical, High, Medium, or Low.

2. Select notification destinations: Email, Slack, Microsoft Teams, ServiceNow, or Webhooks.

3. Enable Notify on Success or Notify on Warning if desired.

4. Set re-notification frequencies (Never, After n failed runs, or Every time).

7. Set execution schedule and persistence

• Scheduling: Run immediately (Run Now) or configure a schedule (Hourly, Daily, Weekly, Monthly, or custom cron).

• Incremental Strategy: Designate a tracking column (timestamp, auto-incrementing ID, or partition) to only process updated records.

• Persistence (Spark only): Store sample rows of valid and invalid records in cloud storage (such as Amazon S3) for debugging.

• Timeouts: Define runtime and total timeout thresholds.

Evaluation rules and scoring logic

Rule evaluation states

• Success: Metric satisfies the expected threshold.

• Warning (optional): Metric indicates degradation but has not crossed the failure boundary.

• Failure: Metric violates the failure boundary.

Threshold directions (absolute evaluation)

• Above: Higher values indicate higher quality (e.g., percentage of non-null records). Default for row-based rules.

• Below: Lower values indicate higher quality (e.g., total error counts).

• Range: Metric must stay between minimum and maximum bounds (e.g., daily row counts).

Policy evaluation strategies

• Rules strategy: The overall policy status matches the lowest status among all individual rules. If any rule fails, the entire policy fails.

• Weightage strategy: Each rule contributes to an overall weighted score (0–100%). The final policy status depends on whether the total score meets the policy-level threshold.

Dynamic column mapping

Apply SQL rules and UDFs across tables with differing schema names using dynamic tags:

1. Use ${column} or ${table} placeholders inside your SQL rule definitions.

2. Apply a shared tag (such as customer_id) to relevant columns across tables (e.g., cust_id, client_id, customer_id).

3. During execution, ADOC automatically evaluates the rule across all tagged columns.

Manual and ad hoc execution

Data Quality policies can be run manually at any time, independent of their schedule, including policies that normally process only incremental data. For execution modes, ad-hoc triggering, overlap handling, and scoring impact, see View Policy Execution and Results.

Execution details and monitoring

After running a policy, view performance metrics in the Policy Execution Details dashboard:

Best practices

• Start with critical baselines: Begin with 3–5 high-impact checks (null checks, unique keys) before adding comprehensive validation suites.

• Calibrate thresholds incrementally: Set initial success thresholds between 80% and 85% based on baseline data, increasing thresholds as quality stabilizes.

• Choose the right engine: Use Pushdown for high-volume, cost-effective scans directly in your warehouse; reserve Spark for multi-table validations or bad-record persistence.

• Isolate test scoring: Enable Skip Scoring during ad hoc or troubleshooting runs to keep baseline reliability trends accurate.