Google Cloud Storage

The gcs reader reads objects from Google Cloud Storage using gs:// roots.

source.yaml

spec:
  secrets:
    gcs_key_id: { type: string }
    gcs_secret: { type: string }
auth:
  provider: gcs
  roots: ["gs://my-bucket/exports"]      # gs:// only
  key_id: "{{ secrets.gcs_key_id }}"
  secret: "{{ secrets.gcs_secret }}"

streams/events.yaml

requests:
  - name: raw_events
    sdk: gcs
    arguments:
      path: "gs://my-bucket/exports/events/*.jsonl"
      format: jsonl

transform:
  mode: page
  steps:
    - name: events
      select: |
        SELECT
          record->>'event_id'                    AS event_id,
          (record->>'timestamp')::TIMESTAMPTZ    AS event_time
        FROM raw_events

export:
  events:
    step: events
    primary_key: [event_id]

Like s3, credentials are a scoped temporary secret, redacted in logs, and roots are contained exactly.