Data Anomaly Policy

A Data Anomaly policy detects unusual values in an asset's profile metrics, such as null percentage, distinct count, and minimum and maximum values. Acceldata Data Observability Cloud (ADOC) learns the normal range of each metric from previous profiling runs and raises an alert when a new profile deviates from it. The policy runs automatically each time the asset is profiled.

Looking for multivariate detection? A Data Anomaly policy evaluates each column's metrics independently. To detect anomalies in the relationships between columns, or to run detection on its own schedule without profiling, use an Auto Anomaly Policy (Preview).

Data Anomaly vs. Auto Anomaly Policies

	Data Anomaly Policy	Auto Anomaly Policy
Policy type in Add Policy	Anomaly	Auto Anomaly (Preview)
Detection method	Univariate: each selected column's profile metrics are evaluated independently	Multivariate: selected columns are analyzed together
When it runs	After each profiling run of the asset	On its own schedule, independent of profiling
Baseline	Previous profiling runs (training window)	Previous runs, or a reference asset
Segmentation	Not available	Segment and subsegment columns
Processing engine	Spark or Pushdown	Spark

When to Use a Data Anomaly Policy

Use a Data Anomaly policy when you need to:

Detect sudden spikes in null values that could indicate a data quality issue.
Identify unexpected changes in distinct count that suggest duplicate or missing data.
Catch unusual minimum or maximum values that point to data entry errors or outliers.
Monitor row count changes that don't follow normal patterns.
Detect gradual degradation in data quality before it affects downstream consumers.

Creating a Data Anomaly Policy

You can create a Data Anomaly policy from Manage Policies or from the Asset Details page.

From Manage Policies

Navigate to Governance > Manage Policies.
Click Add Policy and select Anomaly.
Select the asset to monitor. The Create Data Anomaly Policy page opens.

From the Asset Details page

On the Overview tab, click Actions > Add Data Anomaly Policy.
On the Policies tab, click Add Policy and select Data Anomaly.

The Create Data Anomaly Policy page has five steps.

Step 1: Policy Details

Select the Data Processing Engine: Spark or Pushdown. Click Show Columns to view the asset's columns and continue.

Step 2: Data Columns

Select the columns ADOC includes in profiling and in anomaly detection. By default, all columns are included for profiling. The summary bar at the top of the step shows the total column count and how many are selected for Profiling and Detecting Anomalies.

For each column:

Profile: The column is included in profiling. This is selected for all columns.
Data Anomaly: Select this to monitor the column for anomalies. You must select at least one column.

Use Search by column name to find columns in wide tables. Click Configure Anomaly to continue.

Step 3: Anomaly Configuration

Policy Scoring

Configure when the policy creates an alert.

Setting	Description
Success Threshold (0-100)	The minimum score for the policy to pass. Default: 100.
Warning Threshold (Optional: 0-100)	The score below which the policy raises a warning instead of passing. Default: 70.
Only include anomalies in scoring when the anomaly strength is greater than or equal to	The minimum anomaly strength that counts toward the score: LOW, MEDIUM, or HIGH. Default: LOW. Raise this to ignore weak anomalies.

The score is calculated as the percentage of evaluated metrics that were not anomalous:

Score = ((Total Evaluations − Failed Evaluations) / Total Evaluations) × 100

where Total Evaluations is the number of monitored columns multiplied by the number of profile metrics captured per column, and Failed Evaluations is the number of metrics with a detected anomaly at or above the selected strength.

Example: You monitor 5 columns with 4 profile metrics each, giving 20 evaluations. Two metrics are anomalous, so the score is (20 − 2) / 20 × 100 = 90. With the default thresholds, this run raises a warning.

Alerts & Notification

Setting	Description
Alerts	The severity assigned to alerts raised by this policy. Default: Critical.
Notifications	One or more notification channels to send alerts to. For more information, see [Notification Integrations].
Notify On Success	Also notify when the policy passes.
Notify On Warning	Also notify when the score falls between the warning and success thresholds.
When do you want to be Re-Notified?	Applies while the policy remains in violation across consecutive runs. Reduce Noise: Never notifies once. Reduce Noise: Send notification every N failed policy run re-notifies after every N consecutive failures. Notify Every Time notifies on every run.

Anomaly Detection Settings

Setting	Description
Training Window Minimum	The number of profiling executions ADOC uses to learn normal behavior before it starts identifying anomalies. Default: 7. At least three executions are required, and the training window must span at least 7 days.
Model Sensitivity	How readily the model flags a deviation as an anomaly: LOW, MEDIUM, or HIGH. Default: MEDIUM. Increase sensitivity to catch subtle changes; decrease it to reduce false positives on naturally variable data.

Click Advanced Performance Settings to continue.

Step 4: Advanced Performance Settings (Optional)

These settings control how ADOC loads and processes data during profiling. Keep the defaults unless Acceldata Support advises otherwise.

Processing Batch Size: ADOC loads all columns into memory at once and computes profiling metrics in batches of this size. Default: 30.
Optimize data load with batches: Queries the data in batches instead of loading it all at once, which reduces memory consumption on wide datasets. Batched loading can produce inconsistencies in the profile if the data is updated during profiling.

Click View Summary to continue.

Step 5: Summary

Review the configuration:

Data Columns: Whether anomaly detection is enabled.
Profile Configuration: How many columns are profiled and the profiling type.
Alerts & Notifications: The configured notification channels.

Optionally associate tags with the policy, then click Save Policy.

Executing a Data Anomaly Policy

A Data Anomaly policy does not have its own schedule. It runs automatically each time the asset is profiled, so the profiling schedule determines how often anomalies are evaluated. To run it on demand, profile the asset:

From Manage Policies > Profiles, find the asset and click the Play icon.
From the Asset Details page, click Actions and select the profiling type, Full or Selective.

During the training window, ADOC records profile metrics but does not raise anomalies. Once the training window minimum is met, each profiling run is scored and alerted according to the policy configuration.

Example

A customer table has been reliable for years, but analytics reports begin showing incomplete customer profiles. On investigation, the null percentage in the phone number column has jumped from 5% to 35%. An upstream integration change dropped phone numbers during loading, and the problem went unnoticed for weeks because row counts stayed normal.

A Data Anomaly policy on the phone number, email, and customer ID columns would have learned the normal null percentage during the training window and raised a Critical alert on the first profiling run after the change, within hours rather than weeks.

Best Practices

Start with a few critical columns. Monitor 5–10 business-critical, frequently joined, or low-null columns first and expand once you understand the alert volume.
Match the profiling schedule to the data. Daily loads need daily profiling; weekly loads need weekly profiling. Anomaly detection is only as current as the last profile.
Allow the training window to complete. Review the first anomalies manually before relying on the policy, and expect some false positives while the model learns seasonal patterns.
Tune sensitivity from the default. Start at MEDIUM. Increase it if real issues are missed; decrease it, or raise the minimum anomaly strength for scoring, if alerts are too frequent.
Investigate root causes. A null-percentage spike usually points to an upstream failure; a distinct-count drop to duplicates or data loss; an out-of-range minimum or maximum to data entry errors or schema changes.

Troubleshooting

Symptom	What to check
Too many false positives	Lower Model Sensitivity, raise the minimum anomaly strength for scoring, increase Training Window Minimum, or remove highly variable columns from monitoring.
Real anomalies are missed	Raise Model Sensitivity, confirm the affected columns are selected under Data Anomaly, and confirm profiling runs often enough to capture the change.
Anomalies on every run	The data may be genuinely unstable, or the training window may have covered an abnormal period. Consider a threshold-based Data Quality rule instead of anomaly detection for that metric.
Anomaly detected but the cause is unclear	Compare with Data Drift and Schema Drift policy results on the same asset, and check lineage for upstream changes.

What's Next

Auto Anomaly Policy – Multivariate anomaly detection that runs on its own schedule.
Data Drift Policy – Detect changes in column value distributions between profiles.
Profile Assets – Configure scheduled and on-demand profiling for an asset.
Alerts & Notifications – View and manage the alerts raised by policies.