Certified datasets and SQL cleaning on UC tables

Use Unity Catalog to discover, query, and manage certified datasets. In practice that means gold tables with comments, tags (including a certification/quality tag), a clear owner, and lineage back to a pipeline. Dashboards and Genie should point here — not at raw bronze files.

The outline wants data cleaning on Unity Catalog tables in SQL: invalid values and missing data. Typical ANSI patterns:

sqlSELECT
  customer_id,
  NULLIF(TRIM(email), '') AS email,
  TRY_CAST(signup_date AS DATE) AS signup_date,
  COALESCE(country, 'Unknown') AS country
FROM catalog.gold.customers
WHERE customer_id IS NOT NULL
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY customer_id
  ORDER BY updated_at DESC
) = 1;

• TRIM / NULLIF — empty strings to NULL
• COALESCE — fill defaults
• TRY_CAST — invalid types become NULL instead of failing the query
• Windowed ROW_NUMBER — keep latest row per key

When exploring a DataFrame/table in a notebook, preview can show summary statistics for numeric, string, and date columns plus histograms. That is broader than “row count and names” and not a query profile.

Unity Catalog · Delta Lake