Move & transform data declaratively. /zero-code connectors/
StreamWright extracts from 17+ ad APIs, databases, spreadsheets, and object stores using declarative YAML sources, shapes records in-flight with embedded DuckDB SQL, and writes anywhere — without maintaining connector codebases.
- 17+ Sources
- Ads, DBs, Sheets & S3
- DuckDB SQL
- In-flight transformation
- <50MB / 0 JVM
- Runs locally to K8s
# streams/*.yaml (e.g. streams/records.yaml) — requests → DuckDB SQL → export
requests:
- name: raw_records
http:
path: "/v1/entities"
method: GET
params:
since: "{{ config.start_date }}"
tenant_id: "{{ config.tenant_id }}"
pagination:
type: cursor
cursor_param: next_cursor
cursor_path: "$.meta.next_cursor"
transform:
mode: page
steps:
- name: cleaned_records
select: |
SELECT
record->>'id' AS id,
record->>'name' AS name,
record->>'category' AS category,
(record->>'amount')::DOUBLE AS amount,
record->>'status' AS status,
current_timestamp AS _extracted_at
FROM raw_records
WHERE record->>'status' = 'ACTIVE'
export:
records: { step: cleaned_records, primary_key: [id] }
Connect any source. Shape in DuckDB. Write anywhere.
Universal data integration versatility without heavy container daemons or custom Python wrapper code.
APIs, Databases, Cloud Object Stores
StreamWright
Parquet, DuckDB, Singer, dlt & K8s
The 3-step developer loop
A source is simply a directory: source.yaml plus stream definitions. Every pipeline executes the same battle-tested contract.
Declare Sources in YAML
Zero per-connector code
Define inputs, secrets, authentication, and HTTP endpoints in human-readable YAML. StreamWright handles pagination cursors, rate limits, and exponential retries out of the box.
spec:
config: { tenant_id: { type: string } }
auth:
type: bearer
token: "{{ secrets.api_key }}"Shape In-Flight with DuckDB
Embedded vector SQL engine
Reshape, filter, join, and typecast records using standard DuckDB SQL directly in memory before writing. Eliminate dirty data before it incurs expensive data warehouse storage and compute taxes.
transform:
mode: page
steps:
- name: cleaned_records
select: SELECT id, name, amount::DOUBLE FROM rawRun Locally or Scale to K8s
Orchestrate anywhere
Run syncs locally with a single CLI command, stream to any Singer target or dlt destination, or package sources as containers and orchestrate across Kubernetes using Dagster.
$ sw run --source api_service
✔ 1,000 records synced
✔ Exported to DuckLake & Parquet
✔ 0 errors (0.8s elapsed)Everything a connector needs — as configuration
Ship new sources without maintaining, building, or debugging bespoke connector codebases.
Declarative YAML Sources
Define any connector as pure configuration. Input schemas, credentials, pagination cursors, and HTTP defaults reside in version-controlled YAML files.
In-Flight DuckDB SQL
Transform, clean, and enrich records in memory as they stream. Execute full DuckDB SQL queries without dumping messy JSON or paying heavy warehouse fees.
Cross-Network Vocabulary
One pipeline connects to multiple ad networks (Google, Meta, Microsoft, LinkedIn, OpenAI). Normalize disparate campaign metrics into unified canonical dimensions.
100% Singer & dlt Compatible
Stream records directly into any Singer target, DuckDB/DuckLake, Parquet/CSV files, or dlt destination without custom adapter glue.
Zero-Daemon Local CLI
Installs in seconds via pip. No 8GB JVM overhead, no required Docker daemons, and zero database containers just to run an ad-hoc sync.
Production Orchestration
The same YAML source runs on your laptop, inside container pods, or partitioned across Kubernetes nodes with Dagster asset tracking.
Author & validate pipelines with your AI coding assistant
StreamWright includes a native Model Context Protocol server (sw mcp). AI agents can inspect in-code connector contracts, validate YAML schemas, and test DuckDB SQL transforms in real time without syntax hallucinations.
Why engineering teams choose StreamWright
The declarative simplicity of Singer-style pipelines, modernized with embedded DuckDB SQL transforms and AI agent tooling.
Zero Boilerplate vs Heavy Wrappers
Other solutions require writing Java or Python connector classes and maintaining custom container images. StreamWright connectors are pure declarative YAML files.
In-Flight DuckDB vs Post-Load Only
Traditional tools dump unformatted raw JSON into warehouses and bill you for heavy post-load transformation runs. StreamWright reshapes and filters in-flight using fast DuckDB SQL.
<50MB Process vs Heavy Clusters
Run syncs locally in milliseconds. No mandatory container daemons, no background database instances, and no runtime bloat just to extract API data.
Native AI Protocol (MCP) vs Manual Setup
First data connector framework with a built-in Model Context Protocol server. AI coding agents dynamically inspect schemas and validate sources without syntax errors.
| Capability / Feature | StreamWrightModern ELT | Other Solutions | Custom Scripts |
|---|---|---|---|
| Connector Authoring | ✔Declarative YAML (Draft-7 DSL) | Complex Python / Java classes or bulky UI builders | Bespoke Python scripts & custom wrappers |
| In-Flight Transformations | ✔Built-in DuckDB SQL (filter, join, cast) | None (Raw JSON dump; post-load warehouse ELT only) | Manual Pandas / dict slicing |
| Resource Overhead | ✔<50MB process, 0 background daemons | Heavy (8GB+ RAM, Docker clusters, or multiple virtualenvs) | Low runtime, high maintenance debt |
| AI Agent Native (MCP) | ✔Built-in (sw mcp for Copilot, Claude & Cursor) | None (Manual configuration only) | None |
| Cross-Network Vocabulary | ✔Unified canonical schema abstraction | Disparate per-connector schemas & separate pipelines | Manual column reconciliation |
| Output Targets | ✔Singer targets, DuckDB, Parquet, dlt | Tool-specific destinations or rigid target plugins | Hardcoded to specific databases |
| Time to Author New API | ✔15 to 30 minutes in YAML | Days to weeks per connector | Days of bespoke scripting |
Review our full architectural manifesto in the StreamWright architecture documentation.
Build your first declarative connector in minutes
Install the CLI, point a YAML source at your API, and run it locally in seconds — then scale the exact same source to Kubernetes.