Profile Assets

Profiling is the process of examining a dataset to understand its structure and content. It is the first essential step before you apply monitoring rules or policies in ADOC. Without profiling, you may know a dataset exists, but you would not know whether it is reliable or ready for analytics.

ADOC supports profiling for both structured (tables, numeric/text/date columns) and semi-structured data (arrays, structs, nested fields). This ensures that even complex datasets from systems like Snowflake, BigQuery, or S3 can be analyzed and made trustworthy.

Why Profiling Matters

• Baseline Understanding: See row counts, distinct values, null percentages, and update frequency.

• Issue Detection: Identify missing data, anomalies, or unusual distributions early.

• Policy Preparation: Use profiling results to decide which rules (Data Quality, Freshness, Reconciliation, etc.) make sense for that dataset.

Example: If you profile the EMPLOYEES table, you might learn:

• It has 2,976,508 rows.

• Around 5% of email addresses are null.

• Updates happen daily, but not on weekends.

From this, you might apply a Data Quality policy (no missing emails) and a Freshness policy (daily updates).

How Profiling Fits into the Flow

1. Discover Assets: See what datasets exist.

2. Profile Assets: Understand their shape, quality, and anomalies.

3. Apply Policies: Define standards for “good data.”

4. Monitor Continuously: ADOC watches for policy violations and new anomalies.

Without profiling, policies cannot be applied effectively. Profiling is the bridge between discovery and monitoring.

Starting a Profile

To profile an asset:

1. In the Discover Assets page, locate and select the dataset you want to profile (e.g., EMPLOYEES). This opens the Asset Details page.

2. In the Asset Details page, click the Actions button (top right).

3. Select a profiling mode:

   • Full Profile: Scans the entire dataset for complete statistics.

   • Selective Profile: Profiles a sample of data. Faster, ideal for large datasets.

   • Incremental Profile: Profiles only new data since the last run.

Once profiling completes, the Profile tab in the Asset Details page shows the results.

Profile Execution Details

Every profile captures metadata about when and how it was run.

From ADOC v26.7.0, a Download button is available on the Profile tab when profile data exists. Selecting it exports the profile results — including column statistics, patterns, and top values — as a CSV file. The button is not shown for profiles that are in progress, failed, or have no data. Any view-sensitive-data masking applied to your account is preserved in the exported file.

You can also use the Compare Profiles feature to check how values have changed over time.

Column-Level Metrics

Profiling generates statistics per column, based on data type.

Structured Data

Semi-Structured Data

Recommendation: If a column is semi-structured (e.g., Array, Map, or Struct), click Expand to drill into sub-columns and nested fields.

Column Insights

For each column, click its name to explore:

• Column Statistics: See % null, % unique, and distributions (with bar charts).

• Most Frequent Values: Quickly identify dominant entries.

• Detected Patterns: Common string or numeric patterns in the data.

• Anomalies & Trends: Graphs that highlight unusual changes over time.

Anomaly Detection

When profiling runs over time, ADOC uses historical profiling information to identify unexpected changes in your data.

ADOC supports the following types of anomaly detection:

• Column-level anomaly detection: Monitors profiling metrics for individual columns, such as null percentage, distinct count, minimum and maximum values, mean, and standard deviation, and identifies unexpected changes in those metrics.

• Multivariate anomaly detection: Monitors a group of related columns together and identifies unexpected changes in the relationships among those columns. This can surface issues that may not be detected when columns are monitored individually.

For example, if two columns normally maintain a consistent relationship, multivariate anomaly detection can identify when that relationship changes significantly even when the individual columns do not appear anomalous.

To configure anomaly detection for an asset, create or update a Data Anomaly Policy. For configuration details, see Data Anomaly Policy.

Known Limitations

• Complex sub-columns (in arrays, maps, structs) that are not string/numeric/boolean are treated as strings.

• For array columns:

  • Pattern profile & top values show values only (counts not displayed).

  • Non-null counts are not displayed.

• Nested columns currently do not support anomaly detection.

• By default, ADOC supports profiling up to 5 levels deep in nested structures. This can be changed using the PROFILE_DATATYPE_COMPLEX_SUPPORTED_LEVEL setting.