Data Quality Policy

Data Quality policies validate your data against predefined quality rules to ensure accuracy, completeness, and consistency. These policies check individual data points to identify valid and invalid records based on criteria you define.

• Policy: A set of Rules and Rule Sets that define quality expectations for a dataset.
• Rule: A specific condition or check that data must meet to be considered valid (such as null checks, schema consistency, value ranges, uniqueness, or pattern matching).

Common Use Cases

• Verify customer data completeness before sending records to downstream systems.
• Validate that transaction amounts fall within expected numerical ranges.
• Check that email addresses and phone numbers adhere to standard formatting.
• Flag unexpected null values in primary keys or identifiers.
• Enforce business logic constraints (such as preventing future order dates).

Create a Data Quality policy

You can initiate policy creation either from the Policies menu or directly from an asset's detail page.

1. Start policy creation

• From Manage Policies:
  • In the left navigation menu, go to Data Observability Cloud > Governance > Policies.
  • Select Add Policy in the top-right corner.
  • Select Data Quality as the policy type.
  • Choose the target datasets (assets) to monitor.
• From Asset Details:
  • In the Overview tab, select Actions > Add Data Quality Policy.
  • Alternatively, go to the Policies tab and select Add Policy > Data Quality.

2. Configure data selection

Specify which records or columns ADOC evaluates:

• Column-based selection: Select specific columns directly. Best for simple field validations.
• SQL-based selection: Enter a custom SQL query to isolate a subset.
  • Use {{{lower_bound}}} and {{{upper_bound}}} placeholders for incremental runs.
  • Select Validate to verify syntax before continuing.

If you do not use Native SQL, ensure your query syntax matches your underlying data platform (such as Snowflake or BigQuery).

3. Select an execution engine

Choose the compute engine for running policy jobs:

• Spark (default): Executes on ADOC's Spark engine. Recommended for complex transformations, custom resource allocations, and persisting valid/invalid record samples.
• Pushdown: Runs queries directly inside the source database. Recommended for fast checks on large datasets with minimal data movement.

4. Configure quality rules

Add rules manually or apply a rule set to apply a predefined group of rules. For more information, see Rules and Rule Sets. In Rule Definitions, rules are grouped into Row Based and Aggregate rules. Row-based rules evaluate each row of the selected column. Aggregate rules evaluate a value calculated across the asset.

Rule types

Add a rule

1. In Rule Definitions, select a rule type.
2. Optionally, use the dropdown at the top right to assign the rule to a data quality dimension. The built-in dimensions are ACCURACY, COMPLETENESS, CONSISTENCY, OTHERS, TIMELINESS, UNIQUENESS, and VALIDITY. To create a custom dimension, type a name in the Search or add... box.
3. In Column Name, select the column that the rule evaluates.
4. If the rule type supports them, select the good and bad record options that apply. For more information, see Good and bad records.
5. Select a threshold type: Absolute, Relative, or Anomaly Detection. Then enter the thresholds. For more information, see Configuring Relative Threshold.
6. In Rule Name, enter a name for the rule. If you leave it blank, ADOC generates a name.
7. Select Add.

The rule appears in the list of configured rules below the rule definitions. Each rule evaluates one column. To apply the same check to another column, select that column, enter the thresholds and a rule name, and select Add again. For each rule, the list shows its Name, Type, Column, Data Dimension, Threshold Config, Tags, Criteria, and Weight.

Find, sort, and filter rules

Use the controls above the list of configured rules to find a rule in a policy that has many rules. You can combine search, filters, and sorting.

• Search: In Search by rule or column name, enter all or part of a rule name or column name, and then press Enter. The search isn't case-sensitive.
• Filter: In Type or Dimension, select one or more values, and then select Apply Filter.
• Sort: Select the Name, Type, Column, or Data Dimension column header to sort the list. Select the header again to reverse the sort order. You can't sort by Threshold Config, Tags, Criteria, or Weight.

Good and bad records

ADOC classifies each row that a row-based rule evaluates as a good record if it passes the rule, or a bad record if it fails. Aggregate rules evaluate a single metric across the entire dataset and do not classify individual rows, so sample rows are not available for them in execution results. For some rule types, you can control how null and empty values are classified.

• Include null values in good records: Select to classify rows with null values in the selected column as good records.
• Include empty values in good records: Select to classify rows with empty values in the selected column as good records.

To publish good and bad records to external storage when the policy runs, enable Good/Bad Records in Advanced Execution Settings. For more information, see Set execution schedule and persistence.

Edit rule settings

After you add a rule, select its edit icon in the list of configured rules to open the rule settings panel. The panel contains the following settings. To remove a rule, select its delete icon.

Dimension

Use the dropdown at the top right of the panel to assign the rule to a data quality dimension. The built-in dimensions are ACCURACY, COMPLETENESS, CONSISTENCY, OTHERS, TIMELINESS, UNIQUENESS, and VALIDITY. To create a custom dimension, type a name in the Search or add... box.

Each rule type has a default dimension. For example, Uniqueness Check defaults to UNIQUENESS, and Pattern Match and Tags Match default to VALIDITY. If you assign a rule to a different dimension, its results count toward that dimension's score and no longer count toward the default one.

Dimensions only group rule results. ADOC calculates each dimension's score from the rules assigned to it, using the policy's evaluation strategy. Dimensions don't have weights, so changing a rule's dimension doesn't change the overall policy score. Each execution records the dimension that each rule had when it ran, so later changes don't affect past results.

ADOC creates a custom dimension when you save a rule with a dimension name that doesn't exist yet. Custom dimensions are available in every policy in your tenant. Names aren't case-sensitive, and ADOC saves them in uppercase. Anyone who can create or edit a Data Quality policy can add a custom dimension. You can't rename or delete a custom dimension, so check the existing dimensions before you add one.

Basic

Advanced

Set the policy evaluation strategy

Below the list of configured rules, use Policy Evaluation Strategy to control how ADOC marks each policy execution as Success or Failed.

1. Select a strategy:
   1. Weightage: Each rule contributes to an overall weighted score from 0 to 100. The policy passes when its score meets the success threshold.
   2. Rule Based: The policy status matches the lowest status among all rules. If any rule fails, the policy fails.
2. In Success Threshold (0-100), enter the minimum score required for the policy to pass.
3. Optionally, in Warning Threshold (Optional: 0-100), enter a lower limit. Scores between the warning threshold and the success threshold show a Warning.
4. Select Configure Alerts & Notification to continue.

5. Configure alerts and notifications (optional)

• Select an alert severity level: Critical, High, Medium, or Low.
• Select notification destinations: Email, Slack, Microsoft Teams, ServiceNow, or Webhooks.
• Enable Notify on Success or Notify on Warning if desired.
• Set re-notification frequencies (Never, After n failed runs, or Every time).

6. Set execution schedule and persistence

• Scheduling: Run immediately (Run Now) or configure a schedule (Hourly, Daily, Weekly, Monthly, or custom cron).
• Incremental Strategy: Designate a tracking column (timestamp, auto-incrementing ID, or partition) to only process updated records.
• Persistence (Spark only): Store sample rows of valid and invalid records in cloud storage (such as Amazon S3) for debugging.
• Good/Bad Records: Publish the good and bad records from the policy's rules to external storage each time the policy runs. ADOC writes the records to the location defined by the policy's persistence configuration. For more information, see Persistence Configuration.
• Timeouts: Define runtime and total timeout thresholds.

Evaluation rules and scoring logic

Rule evaluation states

• Success: Metric satisfies the expected threshold.
• Warning (optional): Metric indicates degradation but has not crossed the failure boundary.
• Failure: Metric violates the failure boundary.

Threshold directions (absolute evaluation)

• Above: Higher values indicate higher quality (e.g., percentage of non-null records). Default for row-based rules.
• Below: Lower values indicate higher quality (e.g., total error counts).
• Range: Metric must stay between minimum and maximum bounds (e.g., daily row counts).

Policy evaluation strategies

• Rule Based: The overall policy status matches the lowest status among all individual rules. If any rule fails, the entire policy fails.
• Weightage: Each rule contributes to an overall weighted score (0–100%). The final policy status depends on whether the total score meets the policy-level threshold.

Dynamic column mapping

Apply SQL rules and UDFs across tables with differing schema names using dynamic tags:

• Use ${column} or ${table} placeholders inside your SQL rule definitions.
• Apply a shared tag (such as customer_id) to relevant columns across tables (e.g., cust_id, client_id, customer_id).
• During execution, ADOC automatically evaluates the rule across all tagged columns.

Manual and ad hoc execution

Data Quality policies can be run manually at any time, independent of their schedule, including policies that normally process only incremental data. For execution modes, ad-hoc triggering, overlap handling, and scoring impact, see View Policy Execution and Results.

Execution details and monitoring

After running a policy, view performance metrics in the Policy Execution Details dashboard:

Best practices

• Start with critical baselines: Begin with 3–5 high-impact checks (null checks, unique keys) before adding comprehensive validation suites.
• Calibrate thresholds incrementally: Set initial success thresholds between 80% and 85% based on baseline data, increasing thresholds as quality stabilizes.
• Choose the right engine: Use Pushdown for high-volume, cost-effective scans directly in your warehouse; reserve Spark for multi-table validations.
• Isolate test scoring: Enable Skip Scoring during ad hoc or troubleshooting runs to keep baseline reliability trends accurate.