Skip to main content

Architecture

StreamWright has a small core: a runtime that reads a source and runs each stream's fetch → shape → write, a set of connectors, pluggable outputs, and an orchestration layer that schedules runs per user and network.

The layers​

A run, end to end​

Shared vocabulary across networks​

One pipeline runs across many networks by mapping canonical node names to each network's streams:

Canonical nodenetwork_anetwork_bnetwork_c
accountsaccountscustomersaccounts
campaignscampaignscampaignsunsupported
metricsdaily_metricsstatsinsights

One pipeline uses the canonical names on the left. Each network's streams: map provides its own stream for each node, or marks it unsupported — so the same pipeline runs across networks that name (or lack) streams differently.

Separation of concerns​

  • Source = what to pull and how to shape it (declarative).
  • Connector = how to talk to a system (sign-in, paging, retries).
  • Output = where records go (a run-time choice).
  • Orchestration = when and where runs happen (per user and network).

Each layer is independent, so you can add a connector, switch an output, or change scheduling without touching the others.