DuckPipe

TL;DR

I built DuckPipe, a serverless-first pipeline orchestrator you pip install and run as a plain Python file. No scheduler daemon, no metadata database, no broker, and no additional concepts beyond how Python works. The bet: eliminating that standing infrastructure is worth it regardless of how big your data is, and the orchestrator for the job should be a library, not a platform. It even runs, unmodified, inside a browser tab.

Old way: infra first, logic second

The usual data pipelines I’ve encountered are a few Python functions that depend on each other, say pulling yesterday’s raw records, cleaning them, and rolling them up into a number someone downstream actually needs. Sometimes one step is slow enough that you’d want to skip re-running it when nothing changed. The moment that shape shows up, the reflexive industry answer is Prefect, Airflow, or Dagster.

Most of these solutions ask you to set up infrastructure first: a scheduler, a webserver, a metadata database, usually a broker and a worker pool, before the data analysis reaches whoever needs it. That’s a tax, and it’s the same tax whether your job moves a few thousand rows once a month or a billion rows a minute.

A minimal Prefect flow for a job that pulls yesterday’s NYC taxi trips and rolls them up into daily fare totals:

from prefect import flow, task

@task
def extract_trips():
    ...

@task
def daily_totals(trips):
    ...

@flow
def taxi_pipeline():
    trips = extract_trips()
    return daily_totals(trips)

That part reads like plain Python. Running it in production needs more:

deployments:
  - name: taxi-pipeline-prod
    entrypoint: pipeline.py:taxi_pipeline
    work_pool:
      name: k8s-pool
    schedule:
      cron: "0 6 * * *"

• A prefect.yaml deployment block wiring the entrypoint to a work pool, like the one above
• A worker somewhere actually polling that pool
• Its own Dockerfile to build and push, once the job needs a runtime the default image doesn’t have

None of that is wrong for what Prefect is actually built for: multi-tenant scheduling, retries across a real worker fleet, a UI ops can watch without reading logs over someone’s shoulder. It’s a lot of surface area for “roll up yesterday’s trips,” though, and I’ve watched it compound on a real pipeline system we maintain at work:

• A given pipeline gets described up to three times over: once as flow and task code, once more in prefect.yaml as a deployment, and once again in a hand-rolled YAML file naming which flows compose it, because deployments alone don’t let you say “run these flows together as one pipeline”
• That prefect.yaml is past a thousand lines on its own
• Five separate Dockerfiles cover different runtime needs in that same repo
• An entire second repository exists just to build the 16 base images those deployments pull from

That surface area isn’t really about infrastructure, though. It’s concepts competing to solve overlapping problems, with no strong signal for which one to reach for:

• Flows call tasks, and a flow called from another flow quietly becomes a subflow
• Deployments wrap flows, work pools route to workers, blocks store what those need
• In our case, a homegrown YAML format sits above all of it, because none of the official ones covered composing flows together
• Try to sidestep the menu by staying close to plain Python, and the tax shows up from both directions: plain print()/logging needs explicit Prefect configuration to reach its UI, and code written straight against deployment primitives (workers, schedules, etc.) has no easy way to run locally at all either

It’s the same flavour of confusion PHP got a reputation for: several ways to do roughly the same job, print next to echo, with no strong reason to prefer one. The irony: this is exactly the axis “scaling” is supposed to help with. More pipelines should mean more of one shape repeated, not a wider menu of concepts and a bespoke manifest to cover the gap between them.

I watched a related correction play out at the data-engine layer, years earlier. A pipeline I worked on ran as a Spark job on Amazon EMR, where submitting it costed at least 5 minutes for a cluster to spin up before real work even started. When we reworked it with DuckDB, the same workload ran end-to-end, cold start to finished result, on a 256MB AWS Lambda, in under 5 minutes. It never needed Spark, just one well-tuned process.

What the tax actually costs

I wanted that same correction applied one layer up, to orchestration itself. These are two separate corrections that happen to rhyme, not one that implies the other: matching compute to data volume, and matching standing infrastructure to operational need. This post is about the second one. If Prefect’s primitives, deployment manifests, and dedicated images are the wrong amount of infrastructure for “roll up yesterday’s trips,” the fix isn’t a leaner Prefect. It’s not needing Prefect to answer that question at all.

The claim of “matching standing infrastructure to operational need” is easy to state and hard to size. Here’s what it costs, using a job substantial enough to be a believable stand-in for a real workload, not a claim that it’s typical: a full year of real NYC taxi trips (~35M rows), rolled up by borough, run identically as a plain, on-demand function call, whichever platform:

flowchart LR
    t_extract["extract"]
    t_clean["clean"]
    t_join_boroughs["join_boroughs"]
    t_aggregate["aggregate"]
    t_report["report"]
    t_extract --> t_clean
    t_clean --> t_join_boroughs
    t_join_boroughs --> t_aggregate
    t_aggregate --> t_report
    class t_extract success
    class t_clean success
    class t_join_boroughs success
    class t_aggregate success
    class t_report success
    classDef success fill:#d4f7dc,stroke:#2f9e44,color:#1a1a1a

The only variable that changes is what has to be resident before that call happens. Self-hosted Prefect needs 24/7 standing infrastructure, at a minimum one prefect-server and one prefect-worker, next to a Postgres server. Prefect Cloud submits straight to your own serverless infrastructure:

flowchart LR
    subgraph prefect_arm ["Prefect (self-hosted)"]
        direction LR
        standing["prefect-server +<br/>prefect-worker + Cloud SQL<br/>(24/7, always on)"] --> run2["run_pipeline()<br/>(identical code)"]
    end
    subgraph duckpipe_arm ["DuckPipe"]
        direction LR
        trigger["cron / EventBridge"] --> run1["run_pipeline()"]
    end
    class trigger,run1,run2 ephemeral
    class standing standing
    classDef ephemeral fill:#d4f7dc,stroke:#2f9e44,color:#1a1a1a
    classDef standing fill:#ffd8d8,stroke:#c0392b,color:#1a1a1a
    style duckpipe_arm fill:none,stroke:none
    style prefect_arm fill:none,stroke:none

Measured, not estimated, against sourced energy/PUE/pricing figures (full methodology and citations in DuckPipe’s examples/11_sustainability). The job’s own cost is beside the point as it’s identical in both arms. What isn’t identical is what has to stand there before that call happens, unconditionally, regardless of whether the job is a rounding error or a genuine workload:

• Self-hosted Prefect has no native multi-tenant workspace isolation, so Prefect’s own guidance for that gap is to run one instance per team (the same pattern Airflow vendors call “Airflow sprawl”.) That real setup costs 25.2–35.7 kWh/year and $2,761/year per team. At 50 teams, each with their own instance: ~1,260–1,780 kWh and ~$138,000 a year
• Prefect Cloud’s own workspaces solve that same isolation gap without N installs: the same 50 teams, one shared workspace, priced per seat ($1,200/year each) instead of per instance. At an average of 2-3 seats per team, that’s $120,000-$180,000/year total for all 50

That Prefect Cloud number also has a blind spot the self-hosted one doesn’t: the self-hosted figure comes from a real node this example can measure directly, while Prefect Cloud’s control plane runs on infrastructure nobody outside Prefect can see into. For an org that has to account for it, that’s more than an inconvenience. Reporting regimes like the EU’s CSRD are starting to expect SaaS vendors to disclose energy use as part of Scope 3, leaving the customer to estimate a number it can’t measure.

The tax is real either way, self-hosted or managed; DuckPipe’s bet is that most jobs shaped like this one never needed to pay it.

That raises a sharper question: if the savings come purely from removing standing infrastructure, why not skip DuckPipe and just trigger a plain script on a schedule? For exactly one task, fair enough: there’s nothing here for DuckPipe to add. But past one task, that plain script quietly needs everything DuckPipe already does: DAG/ordering, a way to skip what already ran, a record of what happened, some way to see how it’s going. Build that yourself and you’re hand-rolling a smaller, less-tested version of the same problems Prefect/Airflow/Dagster solve. DuckPipe just solves them without a server standing behind it.

New way: just logic to run

Same job, same shape as the Prefect flow up top, minus everything that isn’t logic:

import duckdb
from duckpipe import task, run

@task
def extract_trips():
    return duckdb.sql("select * from read_parquet('trips.parquet')")

@task(cache=True)
def daily_totals(trips=extract_trips):  # `trips=extract_trips` infers the dependency
    return trips.aggregate(
        "date_trunc('day', pickup_at) as day, sum(fare_amount) as total"
    )

if __name__ == "__main__":
    run(__file__)

duckpipe run pipeline.py

That’s the whole surface:

• No Depends(...) marker, no YAML deployment file, no duckpipe init
• Dependencies come from default-argument values: trips=extract_trips means “run extract_trips first, hand me its result,” nothing to import, nothing to declare twice
• A duckpipe.db file appears next to the script holding run history: the entire “infrastructure,” and a plain file you can duckdb duckpipe.db into and query directly
• Every task gets a content fingerprint (its code, its config, its upstream’s fingerprint), so unchanged work is skipped by default, not by an opt-in flag
• A task’s signature is just Callable[..., Any], so passing one task’s output to the next is a plain function call: whatever you return (a DuckDB relation, a Polars LazyFrame, a pandas DataFrame, a plain string, or None) crosses zero serialization boundaries, because the orchestrator never looks at it

DuckPipe doesn’t care what engine your tasks use, only that something durable remembers what happened, which is what DuckDB is doing in the name: the orchestrator’s own memory.

Task organization holds up the same way past a handful of steps, too. A pipeline is a Python module, so splitting tasks across sibling files needs nothing beyond normal Python imports. And when a task’s body needs to run a whole separate pipeline (e.g. a report generated once per category, each sourced from a shared inner extract-clean-aggregate pipeline), that nesting is supported and shows up as a real nested subgraph in the Mermaid diagram, not a black box. My own preference is still a flat DAG wherever one will do, but that’s a style choice, not a ceiling. DuckPipe doesn’t force it, and it doesn’t fall over the moment a pipeline needs more structure than that.

A real DAG from Datapunk, my own multi-engine benchmark project, leveraging DuckPipe for orchestration:

flowchart TD
    subgraph t_suite_01 ["suite_01"]
        t_suite_01__t_extract["extract"]
        t_suite_01__t_derive["derive"]
        t_suite_01__t_aggregate["aggregate"]
        t_suite_01__t_finalize["finalize"]
        t_suite_01__t_extract --> t_suite_01__t_derive
        t_suite_01__t_derive --> t_suite_01__t_aggregate
        t_suite_01__t_aggregate --> t_suite_01__t_finalize
        class t_suite_01__t_extract success
        class t_suite_01__t_derive success
        class t_suite_01__t_aggregate success
        class t_suite_01__t_finalize success
    end
    subgraph t_suite_02 ["suite_02"]
        t_suite_02__t_extract["extract"]
        t_suite_02__t_clean["clean"]
        t_suite_02__t_aggregate["aggregate"]
        t_suite_02__t_extract --> t_suite_02__t_clean
        t_suite_02__t_clean --> t_suite_02__t_aggregate
        class t_suite_02__t_extract success
        class t_suite_02__t_clean success
        class t_suite_02__t_aggregate success
    end
    subgraph t_suite_03 ["suite_03"]
        t_suite_03__t_extract_lookup["extract_lookup"]
        t_suite_03__t_extract_trips["extract_trips"]
        t_suite_03__t_join_aggregate["join_aggregate"]
        t_suite_03__t_extract_trips --> t_suite_03__t_join_aggregate
        t_suite_03__t_extract_lookup --> t_suite_03__t_join_aggregate
        class t_suite_03__t_extract_lookup success
        class t_suite_03__t_extract_trips success
        class t_suite_03__t_join_aggregate success
    end
    subgraph t_suite_04 ["suite_04"]
        t_suite_04__t_extract["extract"]
        t_suite_04__t_derive["derive"]
        t_suite_04__t_aggregate["aggregate"]
        t_suite_04__t_extract --> t_suite_04__t_derive
        t_suite_04__t_derive --> t_suite_04__t_aggregate
        class t_suite_04__t_extract success
        class t_suite_04__t_derive success
        class t_suite_04__t_aggregate success
    end
    subgraph t_suite_05 ["suite_05"]
        t_suite_05__t_compute_bounds["compute_bounds"]
        t_suite_05__t_extract_candidates["extract_candidates"]
        t_suite_05__t_finalize["finalize"]
        t_suite_05__t_compute_bounds --> t_suite_05__t_extract_candidates
        t_suite_05__t_extract_candidates --> t_suite_05__t_finalize
        class t_suite_05__t_compute_bounds success
        class t_suite_05__t_extract_candidates success
        class t_suite_05__t_finalize success
    end
    t_validate_dashboard_json["validate_dashboard_json"]
    t_suite_01 --> t_validate_dashboard_json
    t_suite_02 --> t_validate_dashboard_json
    t_suite_03 --> t_validate_dashboard_json
    t_suite_04 --> t_validate_dashboard_json
    t_suite_05 --> t_validate_dashboard_json
    class t_suite_01 success
    class t_suite_02 success
    class t_suite_03 success
    class t_suite_04 success
    class t_suite_05 success
    class t_validate_dashboard_json success
    classDef success fill:#d4f7dc,stroke:#2f9e44,color:#1a1a1a

What DuckPipe doesn’t do

Three things are in tension here, and no library this size gets all of them at once: zero standing infrastructure, zero new concepts beyond plain Python, and built-in platform-scale operability (a hosted UI, SLA alerting, governance across many teams, and native massive fan-out.)

That’s not a knock on APScheduler. It’s a solid choice when a schedule can hang off an app already running for other reasons (a web server, a bot), no separate cron or standing service needed. It composes with DuckPipe exactly like cron does, just fire duckpipe.run(pipeline) from the job it triggers.

DuckPipe picks the first two on purpose, not pretending it replaces Prefect/Airflow/Dagster outright. The third isn’t abandoned, though: it’s solved by composition, not by reinventing the wheel:

• Trigger/scheduling is whatever already fires things, on a timer or on an event. Timer-based: a plain cron line, a GitHub Actions workflow on a schedule: trigger, a Kubernetes CronJob. Event-based: an S3 ObjectCreated notification invoking a Lambda directly, no custom glue required, which calls duckpipe.run(pipeline) itself or hands off to a dedicated function if the trigger handler is too small for the real work. Either way, the DuckPipe side is the same: duckpipe run pipeline.py from a shell, or duckpipe.run(module) from Python, so there’s no separate “how does DuckPipe get triggered” story to learn on top of whichever trigger your infrastructure already has
• UI and alerting are a SELECT away, not a missing feature. State is a plain DuckDB file, and SELECT * FROM v_run_summary is the dashboard query already; point that same query at a cron job and a webhook and you have alerting. Point db_path at a DuckLake catalog instead of a plain file, and the same query returns real snapshot history instead of just the latest run. Same commands, same code
• Massive fan-out is exactly the thing a heavier orchestrator is already good at, so DuckPipe leans on it instead of competing with it. Prefect’s .map(), Dagster’s dynamic outputs, and Airflow’s dynamic task mapping can each drive thousands of parallel units, with every mapped unit running a small DuckPipe pipeline internally. The outer platform gets the fan-out and the scheduling visibility it’s built for; each unit gets fine-grained, fingerprint-based incrementality the outer platform’s own task-level caching can’t express on its own. A DuckPipe pipeline is just duckpipe.run(module), an ordinary importable Python call, indistinguishable from any function a Prefect @task, an Airflow PythonOperator, or a Dagster @op might wrap

Smaller fan-out doesn’t even need a heavier platform to lean on. A plain Python loop generating uniquely-named task instances does the job today, without a new decorator to learn. That raises an honest question. For the many teams whose real need is a handful of scheduled scripts, modest fan-out, and a way to see what happened, do they need Prefect, Airflow, or Dagster at all? For a growing number of them, probably not. I’m still not building that dashboard into DuckPipe, though. The moment a read-only query over v_run_summary starts growing alert rules or its own scheduler UI, it stops being a query and starts being a platform, not a library I intend it as.

Remark

The bet underneath all of this isn’t “built for small data.” It’s that pipelines don’t need standing infrastructure to be well-organized, incremental, and observable, and the orchestrator that proves it should get out of the way just as easily as it stepped in. If the zero-server experience is as easy as pip install duckpipe and running a file, there’s almost no reason not to reach for it first. That’s the same correction that made a believer out of me one layer down, watching a 5-minute Spark cluster spin-up turn into a 5-minute Lambda run. Scaling out (a distributed cluster, a serverless executor, or just a browser tab) is meant to extend that same core, not to prove you’ve outgrown it.

None of the above needs to be taken on faith. Try the browser example yourself. If it still vibes for you 5 minutes later, pip install duckpipe is the whole next step.