Skip to main content

What is StreamWright?

StreamWright turns a few YAML files into a working data pipeline. You describe what to pull — from an API, a file store, or a database — and StreamWright handles the how: signing in, paging, retries, shaping the data with SQL, and writing it out. No per-connector code, and one command runs any source.

01requests

Fetch

Pull records from an API, file or database. Sign-in, paging and retries come from the source.

02steps

Shape

Reshape records with DuckDB SQL: filter, join across streams, cast types, compute columns.

03exports

Write

Emit to the output you choose: Singer, files, DuckDB/DuckLake, or a dlt destination.

A source is a folder​

my_source/
├── source.yaml          # inputs, sign-in, HTTP/connection defaults
└── streams/
    ├── campaigns.yaml    # one file per stream
    └── metrics.yaml

Each stream runs three steps:

  • Fetch — requests pull records from an API, file or database.
  • Shape — steps transform records with DuckDB SQL.
  • Write — exports emit to the output you choose.

Why StreamWright​

  • Declarative, not code. A connector is configuration — ship new sources without writing or deploying connector code.
  • Real SQL transforms. Shape records with DuckDB SQL: filter, join across streams, cast types.
  • Shared vocabulary. One pipeline runs across many networks that name their streams differently.
  • Run anywhere. The same source runs locally, in Docker, or on Kubernetes — orchestrated per user and network with Dagster.
  • Open outputs. Singer, JSONL/CSV/TSV/Parquet, DuckDB/DuckLake, or any dlt destination.

🚀 Quickstart →

Install the CLI and run your first source.

📐 Concepts →

Sources, streams, shared vocabulary and outputs.

🏗️ Architecture →

How the pieces fit together.