Architecture Performance Polars Execution

Faster Worker Runs: One Fewer Round-Trip, No More Hangs

Flowfile runs heavy compute in a separate worker process. 0.13.1 drops a round-trip from every remote step and puts a hard timeout on a wedged worker, so a run can't hang forever.

By Edward van Eechoud July 12, 2026 5 min read

Flowfile’s own test suite dropped from about 33 minutes to 25 in the 0.13.1 release. No single trick did that. It came out of paying down the cost of a design decision that’s easy to miss when you’re just dragging nodes around: Flowfile runs your transforms in a different process than the one you’re talking to.

Two processes, on purpose

Flowfile's process and execution map: the Core process (FastAPI, port 63578) runs the DAG engine, never collects data itself, and offloads compute to the Worker process (FastAPI, port 63579), which runs the work in spawned subprocesses and passes data back as Arrow IPC.

flowfile_core is the FastAPI service that holds the graph — the nodes, the wiring, the settings objects the canvas edits. It never calls .collect(). It doesn’t materialize a single row. When a node actually needs to run — read a Parquet file, do the group-by, write the output — core hands the Polars plan to flowfile_worker, a separate service that spawns a subprocess to do the work and hold the resulting data in memory.

Two reasons it’s built this way:

Core stays responsive. A heavy collect can’t block the thread that’s drawing your canvas and streaming progress back, because that thread isn’t the one running Polars.
The compute is killable and isolated. The subprocess owns the dataset memory. Hit Cancel and the child dies, memory and all. A segfault in a native code path takes down the child, not the app. It’s the same reason custom nodes run out there.

The cost of that split is that every offloaded step is a conversation between two processes. Core sends the plan; the worker runs it; core asks how many rows came out; the worker answers; core asks for the schema. Each of those is a round-trip over HTTP on localhost — cheap on its own, not free when a flow has dozens of steps and you run it on every edit. And a separate process introduces a failure mode a single process doesn’t have: what if the child wedges — deadlocked, or stuck in a native call that never returns? Core is left polling a subprocess that will never answer.

0.13.1 went at both.

One fewer round-trip

The worker already knows how many rows it produced — it just materialized them. So it now returns the row count alongside the result, and core stops making a separate call to count rows after every remote or external-source step. One conversation shorter, on every offloaded node. On a flow that’s mostly reads and transforms feeding the worker, that’s a round-trip removed per step, and the whole run feels tighter for it.

Libraries load when they’re needed

Every worker subprocess used to pay the import cost of the big cloud and database clients whether the job touched them or not. Those imports are deferred now, to the jobs that actually use them — so spinning up a worker for a plain CSV-to-Parquet job no longer drags in the S3 stack first. Less to import means each subprocess starts sooner, which matters because Flowfile spawns a fresh one per offloaded task.

A wedged worker can’t hang forever

Every call from core to the worker now has a timeout. The short control calls — is it alive, cancel it — fail fast: a five-second connect, a thirty-second read. The long poll that waits on a real result has a wall-clock ceiling, FLOWFILE_WORKER_TASK_TIMEOUT on the worker’s side and FLOWFILE_WORKER_POLL_TIMEOUT on core’s, both an hour by default and both tunable (set either to 0 to disable).

A task that stops signalling — genuinely deadlocked, or stuck in a C call that never returns — gets stopped with a clear error instead of spinning the monitor until you happen to notice the app went quiet. Generous enough that a legitimately long job isn’t killed; finite enough that a broken one doesn’t cost you the afternoon.

The other half of the clock: parallel workers

Cutting round-trips shortens each step. Running steps at the same time shortens the flow. A flow-level Parallel workers setting (1–32, default 4) lets the worker execute independent steps concurrently: four Read-data nodes feeding a Union don’t have to go one after another — they run as one round, the transforms as the next, the union as the last, and a seven-step flow finishes in three rounds instead of seven.

The same seven-step flow at two Parallel workers settings: with one worker the steps run one after another; with four, the reads run together as round one, the transforms as round two, and the union as round three, finishing in three rounds instead of seven.

It applies to Remote runs only. Local execution is always sequential, and at a setting of 1 every step waits its turn. Nothing about this changed in 0.13.1 — it’s the same behavior that was already there — but it’s now documented with a diagram, and it’s the setting that turns the two-process design into actual parallelism rather than just isolation.

The 33-to-25 figure is our test suite, which spawns and polls the worker more or less constantly — a workload built to hammer exactly the thing that got cheaper. Your own flows won’t see that ratio. But every offloaded step is a round-trip lighter now, and none of them can hang the run — and offload stays a switch, FLOWFILE_OFFLOAD_TO_WORKER on by default, that a sub-flow run deliberately bypasses to execute inside the parent’s own process.

Flowfile is open source (MIT) and runs from a single pip install flowfile, the desktop app, or Docker. The repository is where the two-process engine lives if you want to read it. For more on why single-machine is the deliberate anchor rather than a limitation to apologize for, see Big Data Is Dead; for how core stays plan-only while the worker holds the data, Logic Is a Table, Observed from the Other Side.