Delta Lake

The deltalake reader reads Delta Lake tables directly from local filesystems, Amazon S3 (s3://), or Google Cloud Storage (gs://) without requiring an external metastore or running Spark cluster. It is backed by DuckDB's native Delta Lake extension (delta_scan).

Configuration & storage backends

The deltalake connector decouples storage backends into pluggable storage handlers. Configure credentials and access rules for local files, Amazon S3, Google Cloud Storage, or Azure:

Local files

For local files or pre-mounted volumes, cloud credentials can be omitted entirely. You can optionally restrict path access with roots:

source.yaml

spec:
  config:
    data_dir: { type: string, default: "/data/delta" }
auth:
  provider: deltalake
  roots: ["{{ config.data_dir }}"]   # optional path sandboxing

Amazon S3 / S3-compatible

source.yaml

spec:
  secrets:
    aws_access_key_id: { type: string }
    aws_secret_access_key: { type: string }
auth:
  provider: deltalake
  s3:
    region: us-east-1
    key_id: "{{ secrets.aws_access_key_id }}"
    secret: "{{ secrets.aws_secret_access_key }}"

Google Cloud Storage

source.yaml

auth:
  provider: deltalake
  gcs:
    project_id: my-project

Reading tables with scan

The scan method reads specific columns and applies push-down filter conditions:

streams/events.yaml

requests:
  - name: lake_events
    sdk: deltalake
    method: scan
    arguments:
      table_uri: "s3://my-lakehouse-bucket/delta/events_table"
      columns: [event_id, user_id, event_time, amount]
      filter: "event_time >= '2026-01-01'"
      limit: 10000
transform:
  mode: page
  steps:
    - name: events
      select: |
        SELECT record->>'event_id'                              AS event_id,
               record->>'user_id'                               AS user_id,
               (record->>'amount')::DOUBLE                      AS amount,
               record->>'event_time'                            AS event_time
        FROM lake_events
export:
  events:
    step: events
    primary_key: [event_id]

Running queries with query

Run custom SQL expressions against the Delta table using the query method. The target table is referenced as delta_table:

streams/daily_aggregates.yaml

requests:
  - name: daily_counts
    sdk: deltalake
    method: query
    arguments:
      table_uri: "s3://my-lakehouse-bucket/delta/events_table"
      query: |
        SELECT CAST(event_time AS DATE) AS event_date,
               COUNT(*)                 AS total_events,
               SUM(amount)              AS total_amount
        FROM delta_table
        GROUP BY 1