Skip to main content

Quickstart

Install​

Install the core runtime and CLI from PyPI:

pip install streamwright

This gives you the streamwright CLI (with shorthand sw), the YAML runtime and the connector hub — but no connectors yet. Add the ones you need on demand:

sw connectors -ls                               # browse the hub catalog (or: streamwright connectors list)
sw connectors -i postgres                       # install a reader (files, s3, gcs, postgres)
sw connectors -i google_ads                     # install an API connector (pulls in its SDK)
sw connectors -i google_ads meta_ads restapi    # install several at once

Prefer pip directly? Every connector is pip-installable from its repo:

pip install "git+https://github.com/karthick-jaganathan/streamwright.git#subdirectory=connectors/readers/postgres"

Working from a clone? make install sets up the core plus every connector:

git clone https://github.com/karthick-jaganathan/streamwright.git
cd streamwright
make install

Define a source​

A source is a folder with a source.yaml and one file per stream.

source.yaml
spec:
  config:
    account_id: { type: string }
  secrets:
    token: { type: string }
auth:
  provider: openai_ads
  type: bearer
  token: "{{ secrets.token }}"
http:
  base_url: https://api.example.com
streams/campaigns.yaml
requests:
  - name: raw_campaigns
    http:
      path: /v1/accounts/{{ config.account_id }}/campaigns
      method: GET
transform:
  mode: page
  steps:
    - name: campaigns
      select: |
        SELECT record->>'id'     AS id,
               record->>'name'   AS name,
               record->>'status' AS status
        FROM raw_campaigns
export:
  campaigns:
    step: campaigns
    primary_key: [id]

Run it​

# secrets come from STREAMWRIGHT_SECRET_<NAME> env vars or --secrets FILE
STREAMWRIGHT_SECRET_TOKEN='...' streamwright run my_source \
  --set account_id=123 \
  --output jsonl:out

Records land in out/ as one file per export, plus state.json.

Validate before running​

streamwright validate my_source

streamwright validate checks the source, inputs, stream names and options — without fetching anything. It's ideal for editors and CI.

Try it offline

The files reader runs with no credentials against local files — a great way to see the full fetch → shape → write flow without any API access.

Next: learn the core concepts or browse examples.