SageMaker Data Profiles vs DQDL Rules

Quick answer

Do not choose one data-quality mechanism for every job:

• Profiles/analyzers collect statistics without requiring a threshold.
• Anomaly detection compares a statistic with learned history.
• DQDL rules pass or fail explicit expectations.
• Operational actions decide whether to alert, quarantine, continue, or stop.

The useful loop is profile → observe → investigate → define rule → test response → monitor drift.

Comparison

Dimension	Profile/analyzer	Learned anomaly	DQDL rule
Input	Current dataset	Current metric plus history	Current dataset plus expression
Output	Statistic/observation	Outside predicted range	Pass/fail and evaluated metric
Minimum history	None for one profile	At least three points per AWS	None for static threshold
Quality score	No assertion	AWS says anomaly does not affect score	Rule result contributes to score
Best for	Discovery and trending	Unexpected change	Known invariant or contract
Main risk	Metrics without ownership	Bad baseline or false alert	Blind spot outside authored rules

Use profiles first when the range is unknown

Profile new or poorly understood data to learn completeness, uniqueness, distributions, counts, means, and other statistics. Avoid inventing a “good” threshold before looking at legitimate variation.

Profiles still need data classification, retention, cadence, cost, and owner decisions. Collecting every statistic can widen exposure and create noise.

Use anomalies for change, not truth

Anomaly detection is useful when the metric’s expected range changes with trend or seasonality. It can surface a sudden drop in rows, completeness shift, or distribution movement that no one pre-authored.

But a learned baseline is historical behavior, not a business contract. Bad data in history can become normal, and legitimate launches or backfills can look anomalous. AWS notes that anomaly observations require reviewer feedback because detected points can enter subsequent training.

Use rules for explicit obligations

DQDL is the better authority for requirements such as:

• primary key completeness and uniqueness;
• required columns or schemas;
• legal or contractual value ranges;
• freshness deadlines;
• referential integrity;
• minimum volumes needed by downstream systems;
• hard stop conditions.

Version rules with rationale, severity, owner, test cases, and recovery. A rule without an operational response is only a report.

Combine them without duplicate noise

AWS Glue Data Quality gathers a statistic once when both an analyzer and rule cover the same column. Design the review layer so one underlying event does not create several uncoordinated incidents.

For each metric, define:

1. why it is collected;
2. whether the authority is learned or explicit;
3. the alert severity;
4. the human owner;
5. the stop, quarantine, or continue action;
6. the feedback and rule-promotion path;
7. the retention and access policy.

Example progression

A new orders table begins with row count, completeness, uniqueness, and amount-distribution profiles. After representative history:

• anomaly detection watches seasonal row count and amount distribution;
• DQDL requires non-null order IDs, unique IDs, allowed currencies, and nonnegative amounts;
• a Visual ETL block rule stops missing identifiers;
• an unusual but valid promotion spike is acknowledged rather than encoded as a failure.

This separates learned behavior from non-negotiable contract.

Use the catalog profile guide for at-rest tables, the Visual ETL guide for in-transit enforcement, and the readiness checklist before production.

Frequently asked questions

Should I use data profiles or DQDL rules?

Use profiles to collect statistics and explore behavior, anomaly detection to surface unexpected movement, and DQDL rules to assert known requirements. Most production systems need all three in different roles.

Can anomaly detection replace business thresholds?

No. A learned range can normalize bad history or flag valid change. Known legal, financial, schema, freshness, and operational limits should remain explicit reviewed rules.

Can DQDL rules detect every new problem?

No. Rules cover conditions someone defined. Profiles and anomaly observations can reveal unanticipated statistical changes that should be investigated and, when stable, converted into rules.

Official sources

• AWS Glue: Data quality
• AWS Glue: DQDL reference
• AWS Glue: Anomaly detection
• AWS: In-transit versus at-rest evaluations

Source check: August 19, 2026. Recheck supported analyzers, DQDL rules, modes, history, score behavior, feedback, outputs, and operational actions before use.