Azure Blob Storage

The azure_blob reader reads objects (CSV, TSV, JSON, JSONL, Parquet) from Azure Blob Storage containers using azure:// roots.

source.yaml

spec:
  secrets:
    azure_connection_string: { type: string }
auth:
  provider: azure_blob
  roots: ["azure://my-container/exports"]
  connection_string: "{{ secrets.azure_connection_string }}"

Or using storage account credentials:

source.yaml

spec:
  secrets:
    azure_account_key: { type: string }
auth:
  provider: azure_blob
  roots: ["azure://my-container/exports"]
  account_name: "myaccount"
  account_key: "{{ secrets.azure_account_key }}"

streams/events.yaml

requests:
  - name: raw_events
    sdk: azure_blob
    arguments:
      path: "azure://my-container/exports/events/*.parquet"
      format: parquet

transform:
  mode: page
  steps:
    - name: events
      select: |
        SELECT
          record->>'event_id'                    AS event_id,
          (record->>'timestamp')::TIMESTAMPTZ    AS event_time
        FROM raw_events

export:
  events:
    step: events
    primary_key: [event_id]

Credentials become a scoped temporary DuckDB secret and are redacted (***) in logs. Roots are strictly contained within the specified container and prefix.