Orchestration
The orchestration layer runs the streamwright CLI as scheduled pipelines — one job
per (pipeline, network) — using Dagster. It lives in its own
repository, streamwright-orchestration.
One job per pipeline × network
Each job renders the right streamwright run command for its network (using the
shared vocabulary map) and executes it.
Execution modes
The same job can run the streamwright command in three ways, selected by configuration:
| Mode | Runs the command… | Use when |
|---|---|---|
| Subprocess | In a local process | Local dev, small/simple runs |
| Docker | In a container | Reproducible deps, isolation |
| Kubernetes | As a K8s Job/Pod | Scale, scheduling, resource limits |
Because the unit of work is always streamwright run, moving from local to Kubernetes is
a config change — the source, streams and outputs are identical.
Object storage
Runs can read inputs and write outputs to object storage (S3, GCS), so a Kubernetes job needs no local state — state and exports live in a bucket.