DataFuseAI is evaluating DuckDB as an additional execution engine for pipelines that don't need Databricks or Apache Livy: workloads roughly under 100GB, where spinning up a distributed cluster costs more time than the query itself. This is a concept under evaluation, not a released feature.
- DataFuseAI's existing engines are proven at real scale — a published case study processed 60M+ rows on a 2-core, 16GB Databricks machine — but that same infrastructure is disproportionate for small validation runs and exploratory pipelines.
- DuckDB is an in-process, single-node analytical database that queries Parquet, CSV, and JSON directly, including files sitting in S3, without a separate server to provision.
- Independent benchmarks, including a published TPC-DS comparison at roughly 100GB, repeatedly show single-node engines like DuckDB outperforming small or local-mode Spark deployments below that threshold, and hitting real limits above it.
- The concept: add DuckDB as a selectable execution engine alongside Databricks, Livy, and DataFuseAI's native engine, chosen by workload size rather than forced through the same distributed path every time.
- This blog describes the idea and the open questions around it.
What "concept under evaluation" means here. Everything below describes a direction DataFuseAI is exploring, grounded in DuckDB's real, documented capabilities. It does not describe a shipped feature, a beta, or a committed roadmap item. Wherever this blog says "could" or "would," that's deliberate.
The Gap Below the Distributed-Cluster Threshold
DataFuseAI's own 60 million row case study ran on a 2-core, 16GB Databricks machine, with peak throughput measured at 64,439 rows per second. That result says something real about what the platform's existing engines can do at scale. It says nothing about what happens on the other end of the size range: a 40MB CSV someone wants profiled before a meeting, a validation pass against last week's export, a pipeline still being built and re-run a dozen times during testing.
Right now, all of those workloads take the same path as the 60-million-row run: through Databricks, through Apache Livy, or through DataFuseAI's native engine, each of which assumes there's real work worth distributing. For a file that fits comfortably in memory, that assumption doesn't hold. The setup cost of reaching a distributed engine, however fast it eventually runs, can exceed the entire runtime of the query itself.
That gap, between what DataFuseAI's engines are built for and what a meaningful share of real pipeline traffic actually looks like, is the problem this concept is trying to answer.
Why DuckDB Specifically
DuckDB is an in-process analytical database: it runs embedded inside whatever process calls it, with no separate server to install, and it executes SQL directly against Parquet, CSV, and JSON files rather than requiring data to be loaded into a proprietary store first.[5] That architecture matters more than it sounds like it should. A distributed engine's overhead comes largely from coordination: scheduling work across executors, shuffling data between nodes, managing a cluster that has to exist before the first query runs. An in-process engine skips most of that by design, because there's nothing to coordinate.
The published evidence for where this actually pays off is more specific than "DuckDB is fast." A data engineering team ran the TPC-DS benchmark at a scale factor of 100, roughly 100GB of input data, comparing dbt paired with DuckDB against three different Spark configurations on identical AWS instances. DuckDB came out consistently faster across most of the query set.[3] It also hit a real wall: the single heaviest query in the benchmark, a join across the two largest fact tables, ran out of memory and failed, even on larger instances.[3] That's the honest shape of the tradeoff. Single-node engines win comfortably on workloads that fit the description "low to medium volume." They lose, predictably, once a query's working set outgrows one machine.
A separate benchmark from an ad-tech engineering team compared DuckDB against Spark and ClickHouse's embedded variant for a small-scale pipeline, and reached a similar conclusion from a different angle: DuckDB is single-node by design, which caps how far it scales, but for datasets that comfortably fit on one machine, its speed and simplicity make it the more practical choice.[4] Neither of these are DataFuseAI's own numbers. They're independent, named, published comparisons, and they're consistent with each other.
What This Could Look Like Inside DataFuseAI
Consider a pipeline built today against a mid-sized PostgreSQL table: a filter, a join against a lookup table, an aggregation, written to a Parquet file for a BI tool to pick up. Under the current model, that pipeline reaches Databricks or Livy to run, the same as a pipeline processing hundreds of millions of rows. In the concept being evaluated, the same pipeline definition could instead route to a DuckDB engine option sitting alongside Databricks, Livy, and DataFuseAI's native engine in the existing engine selection, chosen automatically below a size threshold, or set explicitly by whoever builds the pipeline.
The technical building blocks for this already exist inside DuckDB itself, independent of anything DataFuseAI would need to build from scratch. Its httpfs extension queries Parquet and CSV files sitting in S3 directly, using HTTP range requests to read only the bytes a query actually needs rather than downloading a full file first.[1] Its PostgreSQL extension attaches directly to a live Postgres database and can read tables in place without a separate export step.[2] Both map cleanly onto source and sink types DataFuseAI's connector model already understands. Neither has been wired into DataFuseAI. Both are the reason this concept is more than a slide.
None of this would touch how a pipeline is designed. The drag-and-drop canvas, the transform library, the way sources and sinks get configured all stay the same. What changes is which engine actually executes the compiled result, and that choice is exactly the kind of decision DataFuseAI already treats as separate from pipeline design for Databricks, Livy, and its native engine. Adding DuckDB would extend a pattern that already exists rather than invent a new one.
Where the Line Would Sit
A DuckDB engine option would not replace anything DataFuseAI already runs. It would sit underneath the workloads currently overserved by a distributed engine, not compete with the ones that need one. The distributed processing architecture DataFuseAI runs on today exists specifically because some pipelines genuinely need to shuffle data across multiple machines, and no single-node engine, DuckDB included, changes that math.
Sub-100GB files or tables. Profiling and validation passes. Pipelines still being built and tested. Local development against sampled data.
Multi-hundred-gigabyte transformations. Heavy shuffle joins across large fact tables. Production workloads already proven out on Databricks or Livy, like the 60M-row case study above.
The size line isn't a fixed number. It moves with query complexity, available memory, and how much of a dataset a given transform actually has to hold at once. That's the honest version of the boundary, not a marketing threshold, and it's why any engine-routing logic would need real testing against DataFuseAI's own pipeline patterns before shipping, not a number borrowed from someone else's benchmark.
Frequently Asked Questions
No. This describes a concept under evaluation, not a released capability. Nothing described here is available in the product yet.
No. The concept adds a lightweight option alongside the existing engines for workloads that don't need distributed compute. Databricks and Livy would remain the path for large-scale or production workloads.
There's no fixed number. Published independent benchmarks put single-node engines like DuckDB ahead of small-cluster or local-mode Spark deployments up to roughly 100GB, depending heavily on query complexity and available memory, with real limits appearing above that on heavy joins.
Yes. DuckDB's httpfs extension queries Parquet and CSV files directly from S3 and compatible object storage without downloading them first, which is one of the reasons this concept is technically plausible rather than purely theoretical.
Not directly, though the timing raised the topic's visibility. AWS's acquisition validated the broader idea that lightweight, single-node engines matter for a meaningful share of real workloads. This concept is about what that idea means specifically for DataFuseAI's own engine roster, independent of who owns DuckDB's maintainers.[6]
