Amazon S3

The s3 reader reads objects (JSONL, CSV, Parquet) from Amazon S3. Credentials are secrets; the rest are settings.

source.yaml

spec:
  secrets:
    aws_key_id: { type: string }
    aws_secret: { type: string }
auth:
  provider: s3
  roots: ["s3://my-bucket/exports"]      # buckets/prefixes the reader may read
  key_id: "{{ secrets.aws_key_id }}"
  secret: "{{ secrets.aws_secret }}"
  session_token: "{{ secrets.aws_session_token }}"   # optional
  region: us-east-1
  endpoint: "s3.amazonaws.com"           # optional (host[:port])
  url_style: vhost                       # vhost | path
  use_ssl: true

streams/events.yaml

requests:
  - name: raw_events
    sdk: s3
    arguments:
      path: "s3://my-bucket/exports/events/*.parquet"
      format: parquet

transform:
  mode: page
  steps:
    - name: events
      select: |
        SELECT
          record->>'event_id'                    AS event_id,
          (record->>'timestamp')::TIMESTAMPTZ    AS event_time
        FROM raw_events

export:
  events:
    step: events
    primary_key: [event_id]

Credentials become one temporary scoped secret and are *** in logs. Roots are contained exactly (s3://acme/ does not match s3://acme-evil/…).