Auto Anomaly Policy

Preview

Auto Anomaly policies are available in Preview in ADOC 26.8.

An Auto Anomaly policy uses multivariate anomaly detection to find unexpected patterns in an asset, including anomalies that only appear in the relationships between columns. It runs a set of predefined anomaly checks on a schedule you control. Unlike a Data Anomaly policy, which evaluates profile metrics after the asset is profiled, an Auto Anomaly policy runs as an independent job with its own execution schedule, incremental strategy, and Spark resource settings, and can compare each run against a reference asset instead of historical baselines.

Auto Anomaly vs. Data Anomaly Policies

Acceldata Data Observability Cloud (ADOC) provides two anomaly policy types. Use this table to choose the right one.

	Data Anomaly Policy	Auto Anomaly Policy
Policy type in Add Policy	Anomaly	Auto Anomaly (Preview)
Detection method	Univariate: each selected column's profile metrics (null %, distinct count, min/max, and so on) are evaluated independently	Multivariate: selected columns are analyzed together, so anomalies in relationships between columns are detected
Checks	Profile metric deviations	Predefined anomaly checks (Value Deviation, Distribution Shift, Missing Rate, and so on), each optionally scoped to columns

	Data Anomaly Policy	Auto Anomaly Policy
When it runs	After each profiling run of the asset	On its own schedule, or on demand [TO CONFIRM: confirm on-demand run from Manage Policies is supported]
Baseline	Historical profile runs (training window)	Historical runs (minimum training runs), or a reference asset
Segmentation	Not available [TO CONFIRM: confirm]	Segment and subsegment columns
Processing engine	Spark or Pushdown	Spark [TO CONFIRM: confirm no engine selector is shown; only Spark resource settings]

How Multivariate Anomaly Detection Works

Rule-based Data Quality policies and single-column anomaly checks catch known problems. They miss issues that only appear in the relationships between columns, for example a payment method that is normal overall but anomalous for one product category, or a logging change that shifts the joint distribution of several behavioral fields at once.

An Auto Anomaly policy uses multivariate anomaly detection: it analyzes the selected columns together, learns the normal relationships between them, and flags rows and columns that deviate from those relationships. Each run:

Reads the asset in full or incrementally, depending on the execution settings.

Builds or updates the baseline from previous runs, or reads the reference asset if one is configured.

Runs the enabled checks against the current data.

Records the anomalous rows, columns, and segments, and raises alerts according to the notification settings.

Scoring: The policy score is the percentage of rows scanned in the run that were not flagged as anomalous.

Severity Scoring

Each finding raised by a check carries an effect size of Small, Medium, or Large, based on how far the data deviates from the expected baseline. Within a run, all findings for a segment are combined into a single severity level, shown as None, Low, Medium, High, or Critical. Larger and more numerous findings push the segment's severity higher. This is the severity used to color-code results on the execution details page, for example under Top Anomalies.

When to Use an Auto Anomaly Policy

Use an Auto Anomaly policy when you need to:

Detect anomalies without configuring a profiling schedule for the asset.
Compare each run against a known-good reference asset rather than waiting for a training period.
Analyze anomalies within segments of the data, for example per region or per product line.
Run anomaly detection incrementally on new data only, or on multiple schedules aligned to your pipeline cadence.

Creating an Auto Anomaly Policy

Navigate to Governance > Manage Policies.
Click Add Policy and select Auto Anomaly.
Select the asset to monitor.

ADOC names the policy automatically in the format <asset name>-auto-anomaly-policy-<policy ID>, for example CUSTOMER_ORDERS-auto-anomaly-policy-84789420.

The configuration wizard has five steps. Only Detection Configuration is required; the remaining steps are optional and use default values if you skip them.

Step 1: Detection Configuration

Enable auto anomaly detection and define how the baseline is established.

Field	Description
Auto Anomalies	Turn on to enable anomaly detection for this policy.
Segment Columns (optional)	Columns used to split the data into segments. Anomaly checks are evaluated per segment.
Subsegment Column (optional)	A column used to further divide each segment. Defaults to None.
Reference Asset (optional)	When set, ADOC reads this asset on every run and uses its statistics as the baseline instead of historical run averages. Training runs are skipped. Click Select Asset to choose it.
Minimum Training Runs	The number of initial runs that only collect baseline statistics before ADOC starts flagging anomalies. Default: 4. Not used when a reference asset is set.

Click Configure Checks to continue.

Step 2: Configure Checks

Choose which anomaly checks the policy runs and, optionally, which columns each check applies to. All checks are enabled by default.

When no columns are selected, most checks apply to all columns of the applicable type. This is confirmed for two checks: Association Patterns analyzes all categorical column pairs automatically unless specific pairs are added, and Categorical Outlier defaults to all detected categorical columns.

Check	Detects
Value Deviation	Flags individual rows whose numeric value is a statistical outlier, far from the expected range for that column. Useful for catching sensor spikes, pricing errors, or latency timeouts caused by a single anomalous record.
Distribution Shift	Detects when the overall shape of a numeric column's distribution changes, even if no single value is an extreme outlier. Useful for catching regime changes, such as a new pricing floor or a pipeline that starts dropping values from part of the expected range. Requires a historical baseline; the check does not run until baseline data has been collected.
Stability	Detects when a numeric column becomes significantly more volatile than it used to be, even if its average value hasn't shifted. Useful for monitoring manufacturing tolerances, financial spreads, or network jitter, where consistency matters as much as the values themselves.
Peer Outlier	Detects when one segment's values are a statistical outlier compared to its peer segments, for example one region, store, or device behaving very differently from the others even though its values fall within a normal overall range. Requires at least three segments.
Missing Rate	Detects columns with an unexpectedly high rate of null values. Useful for spotting upstream pipeline issues such as broken joins, incomplete API responses, or schema mismatches that cause a field to stop populating.
Schema Stability	Detects when an entire column contains no values in a run, effectively disappearing from the data. Useful for catching dropped columns, renamed fields, or upstream schema changes that haven't been propagated. Unlike Missing Rate, which flags partially missing data, Schema Stability flags a column that is completely empty.
Frequency Shift	Detects unexpected changes in the distribution of a categorical column's values, for example a new status code appearing or the mix of payment methods shifting significantly. Useful for monitoring enum-like fields, status codes, or country codes where the set and proportion of values should stay stable.
Association Patterns	Detects when the expected pairing between two categorical columns breaks down, for example a product that has always shipped from one supplier suddenly appearing with a different one. Useful for validating relationships that aren't enforced by the schema, such as product-supplier pairings or route-carrier assignments. Click + Add pair to restrict the check to specific column pairs; by default, all categorical column pairs are analyzed automatically.
Version Consistency	Detects gaps or regressions in a sequential integer column, such as Kafka offsets, auto-increment IDs, or batch counters. Useful for message queues, event streams, or audit logs where values are expected to increase without skipping.
Contribution	Detects when a segment's share of the total row count is unexpectedly high or low, which can indicate a data pipeline imbalance. Useful for multi-tenant datasets or tables partitioned by category or region, where each segment is expected to contribute a roughly predictable proportion of records.
Segment Drift	Detects when a segment that appeared in the historical baseline is missing from the current run, or when a new segment appears that wasn't present during training. Useful for monitoring datasets partitioned by a category column, such as country or product category, where the set of active values should stay stable. Requires a historical baseline and Segment Columns to be configured.
Categorical Outlier	Flags individual rows whose categorical value is inconsistent with the context set by other columns in the same row, the categorical equivalent of Value Deviation. Useful for detecting miscoded records, data entry errors, or values assigned to the wrong category, such as an unexpected status for a given order type.

To exclude a check, turn off its toggle. Click Configure Alerts & Notifications to continue.

Step 3: Alerts & Notifications

Define how ADOC notifies you when the policy detects anomalies.

Severity: The severity assigned to alerts raised by this policy. Default: Critical.
Notification Channels: One or more configured channels to send alerts to. For more information, see Notification Integrations.
Notify On Success: Turn on to also send a notification when a run completes without anomalies.
When do you want to be Re-Notified?: Applies when the policy stays in violation across consecutive runs.
- Reduce Noise: Never — Notify once; do not re-notify while the violation persists.
- Reduce Noise: Send notification every N failed policy run — Re-notify after every N consecutive failed runs.
- Notify Every Time — Notify on every run that detects anomalies.

Click Configure Execution Details to continue.

Step 4: Configure Execution Details

Incremental Strategy and Scheduling

Run this policy incrementally: Turn on to evaluate only new or changed data on each run, using an incremental strategy.
Schedule policy execution: Turn on to run the policy on a schedule. You can select an existing schedule or create a custom one. Auto Anomaly policies support up to seven independent schedules with a shared time zone. For more information, see Policy Scheduling.

Advanced Execution Settings (optional)

Resource Strategy: The Spark driver and executor resources for the job. Choose Small, Medium (default), Large, or Custom to define your own limits. Click View Details on a tier to see its full specification.
Auto-retry with higher strategy: If the job fails with an out-of-memory error, ADOC automatically restarts it with the next resource tier (for example, Small to Medium).
Additional Spark Configuration: Spark properties to apply to the job, entered as key-value pairs.
Yunikorn Queue Name: Overrides the default Yunikorn queue for this job.
Persistence Configuration: Where the policy writes its results. Defaults to SYSTEM DEFAULT. For more information, see Persistence Configuration.
Bad Records and Good Records: For each, choose whether to Persist the records and whether to persist a Sample Only. By default, bad records are persisted as a sample only and good records are not persisted.

Click View Summary to continue.

Step 5: Summary

Review the detection configuration, checks, alerts, and execution settings, then save the policy.

Executing an Auto Anomaly Policy

Auto Anomaly policies run on the schedules configured in Step 4.

During the first runs (up to Minimum Training Runs), the policy collects baseline statistics and does not raise anomalies. If a Reference Asset is set, anomalies can be raised from the first run.

Viewing Results

Open a policy execution from Manage Policies or from the alert to see:

Rows scanned out of the total rows in the asset, and columns analyzed out of the total columns.
Anomalies detected, as a count and as a percentage of scanned rows.
Anomalous columns, with an explanation of why each column was flagged.
Top anomalies: the highest-scoring anomalous rows, color-coded by severity, with the contributing columns highlighted.
Anomalous segments (when segment columns are configured): which segments contribute the most anomalies, in absolute terms and relative to the segment's row count.
Metric trends with anomalies highlighted, and the earliest timestamp at which anomalies were detected when the asset has date or datetime columns.

Managing Auto Anomaly Policies

In Manage Policies, use the Auto Anomaly filter to list only Auto Anomaly policies. The Policy Type column shows Auto Anomaly for these policies and Data Anomaly for policies created with the Anomaly type.

What's Next

Data Anomaly Policy — Profile-metric-based anomaly detection that runs after asset profiling.
Policy Scheduling — Configure single or multiple schedules for a policy.
Alerts & Notifications — View and manage alerts raised by policies.
Segments — Define segment columns for segment-level analysis.