swale

A scheduled, dependency-ordered orchestrator of asset graphs. The deployment consists of a single binary and a single bucket.

swale runs on the taquba durable execution crates. Every task instance runs as a single workflow run, and the record of its outcome and the event for the scheduler commit in one transaction. The swale crate is the library and the binary, and its README documents the API and the command in full.

One TOML file per graph. An asset node declares the asset it produces and the assets it consumes, and a task node declares its upstream nodes with after. Every node runs an operator with parameters, and a parameter string can refer to the partition, to the output of an upstream node or to an environment variable of the daemon.

[graph]
name = "orders_daily"
schedule = "0 2 * * *"
partition = "daily"

[[node]]
name = "extract"
produces = "orders_raw"
operator = "shell"
[node.params]
command = "dbt run --select orders_raw >&2 && printf '{\"table\": \"orders_raw\"}'"

[[node]]
name = "load"
produces = "orders_warehouse"
consumes = ["orders_raw"]
operator = "http"
[node.params]
method = "POST"
url = "https://warehouse.example/load"
body = '{"table": "{{ upstream.extract.table }}"}'

swale has four built-in operators:

• subprocess runs a program with a JSON document on stdin.
• shell runs a command line with sh -c.
• http sends one request.
• object_exists waits for an object at a store URL.

The output of a node is the JSON it prints or receives, and a downstream node refers to it as {{ upstream.<node>.<path> }}.

swale validate checks a definition and reports every fault at once. swale run runs a graph for one partition on a store, a directory or an object store URL (s3://, gs://, az://), and resumes an existing graph run.

swale publish writes a definition to the store, and swale daemon runs the published graphs: it fires each schedule, catches up a new graph and replays the firings missed during downtime.

$ cargo install swale
$ swale validate swale/examples/orders_daily.toml
orders_daily: 6 nodes, 4 edges
$ swale run swale/examples/local.toml --store ./swale-store
local/none: started, 1 root node(s) submitted
  first: succeeded (local-none-first-r0)
  second: succeeded (local-none-second-r0)
  third: succeeded (local-none-third-r0)
local/none: complete
$ swale publish swale/examples/orders_daily.toml --store s3://bucket/swale
orders_daily: published ac04ecabf07dc9a6d74c698a89e90f5d36c0da5ae659b6bcf7c81f4880590628
$ swale daemon --store s3://bucket/swale --pool warehouse=2

Licensed under either of Apache License, Version 2.0 or MIT license at your option.