REST API

The restapi connector enables declarative stream pipelines for any REST API service (GitHub, Stripe, Shopify, Jira, webhook feeds, internal microservices) without having to create a separate bespoke connector package.

It executes requests directly via StreamWright's declarative HTTP Transport engine.

Installation

Install the package standalone via the StreamWright CLI:

streamwright connectors install restapi

Or with pip:

pip install streamwright-restapi

Source Configuration

Configure your API credentials and default HTTP settings in source.yaml. You can use any of StreamWright's supported authentication schemes (bearer, api_key, basic, or oauth2_refresh_token).

source.yaml

kind: source
name: github_demo

spec:
  config:
    org: { type: string, description: "GitHub organization name" }
  secrets:
    github_token: { type: string, description: "Personal access token" }

auth:
  provider: restapi
  type: bearer
  token: "{{ secrets.github_token }}"

http:
  base_url: https://api.github.com
  headers:
    Accept: application/vnd.github+json
    X-GitHub-Api-Version: "2022-11-28"
  paginator:
    type: link_header
  retry:
    codes: [429, 500, 502, 503, 504]
    max_attempts: 5
    backoff: exponential

All credentials and tokens are automatically treated as secrets and redacted as *** across all logs.

Defining Streams

Streams define declarative HTTP requests under requests[].http and transform extracted JSON records using DuckDB SQL:

streams/repositories.yaml

description: List repositories in an organization.

requests:
  - name: raw_repos
    http:
      path: "/orgs/{{ config.org }}/repos"
      method: GET
      params:
        per_page: 100
        sort: updated

transform:
  mode: page
  steps:
    - name: repos
      select: |
        SELECT record->>'id'            AS id,
               record->>'name'          AS name,
               record->>'full_name'     AS full_name,
               (record->>'private')::BOOLEAN AS is_private,
               (record->>'stargazers_count')::INTEGER AS stars
        FROM raw_repos

export:
  repositories:
    step: repos
    primary_key: [id]

Stream Requests Reference

Stream requests support the full power of StreamWright's declarative HTTP engine:

• Relative & absolute URLs: endpoints relative to http.base_url or complete URLs.
• Dynamic templating: Jinja expressions referencing config.*, secrets.*, partition.*, and window.*.
• Query params & JSON payloads: params, json, or raw data.
• Built-in pagination: offset, cursor, page_number, link_header, or none.
• Automatic retries: exponential backoff on HTTP 429 and 5xx responses.