Skip to main content

Sources

A source is a folder that defines a connector as data. It has one source.yaml and a streams/ folder with one file per stream.

my_source/
├── source.yaml
└── streams/
    ├── accounts.yaml
    ├── campaigns.yaml
    └── metrics.yaml

source.yaml​

source.yaml declares the source's inputs, how to sign in, and the defaults every stream shares.

source.yaml
spec:
  config:                       # non-secret, per-client settings
    account_id: { type: string }
    region:     { type: string, default: us }
  secrets:                      # credentials, never written in the source
    token: { type: string }
auth:
  provider: openai_ads          # or linkedin_ads, amazon_ads, google_ads, postgres, ...
  type: bearer                  # bearer, oauth2_refresh_token, api_key, basic
  token: "{{ secrets.token }}"
http:                           # shared HTTP defaults for REST sources
  base_url: https://api.example.com
  paginator: { type: cursor, token_path: next_token, param: cursor }
  records: { path: data }
  retry: { codes: [429, 500, 502, 503], max_attempts: 5 }
  • spec.config — values a client provides per run (via --set, a --config file, or defaults). Referenced as {{ config.NAME }}.
  • spec.secrets — credentials supplied from STREAMWRIGHT_SECRET_<NAME> environment variables or a --secrets file. Referenced as {{ secrets.NAME }} and redacted in every log line.
  • auth — the connector and authentication method (typed authenticators such as bearer or oauth2_refresh_token, or an SDK connector like google_ads, postgres, s3).
  • http — shared HTTP defaults for REST sources (base_url, paginator, records, headers, retry; see HTTP Transport).

Client settings​

The same source serves many clients. Each client's run supplies its own config and secrets — nothing client-specific lives in the source:

streamwright run my_source \
  --config clients/acme.yaml \     # this client's config + which streams to run
  --secrets clients/acme.secrets.yaml

This separation is what lets one source definition run for every client, and one pipeline run across many networks (see shared vocabulary).