Case Study: How DataFuseAI Processed 60 Million Rows Without a Single Line of Code

Every Platform Claims It Scales. We Decided to Prove It.

Every data platform says the same thing. In every pitch deck, every landing page, every sales call: “We handle any data volume. We grow with you.”

You’ve heard it before. The real test comes when that promise meets reality. When 60,000 rows become 6 million, and then 60 million. When your joins get heavier, your transformations more complex, and your infrastructure more constrained.

Ask any vendor to publish exactly how long their platform takes to move and transform data at that scale. Ask them to break it down by phase, by pipeline type, by compute environment. Watch how fast the conversation shifts to it depends and let’s schedule a follow-up.

Scaling a data pipeline is easy in a demo. Proving it is another matter entirely—and proof is what most platforms quietly avoid.

We decided to stop saying it and start showing it.

So we ran the benchmark ourselves: 31 pipeline runs, on a deliberately modest machine, using an industry-standard dataset. We are publishing everything. Every second, every phase, every bottleneck, every constraint. No curation. No convenient rounding. Actual, transparent, reproducible results that show what happens when you push real volume through the system and keep pushing.

This is what happened.

Benchmark Methodology

Dataset: TPC-H (Industry Standard)

The benchmark used TPC-H, the Transaction Processing Performance Council’s decision-support dataset, widely considered the gold standard for testing analytical data workloads. TPC-H isn’t a toy dataset or a synthetic stress test. It simulates what real businesses actually run: aggregations, multi-table joins, filters, and derived calculations at scale. It’s what the industry uses when the goal is honest, comparable measurement.

The focus was on the two heaviest TPC-H tables—lineitem and orders—tested across four volume tiers:

That’s a 1,000× spread from smallest to largest. The same pipelines, the same platform, across the full range. No shortcuts, no cherry-picked configurations.

Compute: A Realistic, Everyday Setup

This wasn’t run on an enterprise-grade cluster with 64 cores and a terabyte of memory. It ran on the kind of modest setup most smaller teams actually work with day to day:

The choice was deliberate. We wanted to answer a specific question: what happens when you run serious data volume on the kind of hardware that most smaller teams actually have? If DataFuseAI performs at this level on constrained infrastructure, production deployments on properly scaled environments will have real headroom to go further and produce a higher performance.

8 Pipeline Configurations

The benchmark ran across eight distinct pipeline types, covering the transformation patterns most common in real production work. Each one tests a different combination of operations that data teams deal with daily:

These aren’t theoretical patterns. They’re the building blocks of virtually every ETL workflow in production—from simple single-table rollups to complex multi-source, multi-destination orchestrations.

What the Numbers Show

Sub-Linear Scaling: The Headline Result

The most important finding from this benchmark: data volume grew 1,000×, but total processing time grew just 50×.

At 60 million rows, the fastest single pipeline completed in 15.5 minutes. Peak single-run throughput hit 64,439 rows per second—on two cores.

This sub-linear relationship between data and time is what separates platforms actually designed for scale from platforms that just describe themselves that way. Linear scaling means every 10× increase in data requires a 10× increase in time (and often cost). Sub-linear scaling means the platform gets proportionally more efficient the harder you push it.

How Long Each Scale Actually Takes

Ten times the data. Eight more seconds. That’s what happens from sf001 (60K rows) to sf01 (600K rows). The platform absorbs a full order of magnitude in additional volume with barely a pause. The real climb starts at sf1 (6M rows), and by sf10 (60M rows) total time has grown—but nowhere near proportionally to the data.

The Spread Within Each Scale

Average time only tells part of the story. The range between the fastest and slowest pipeline at each scale reveals where the real variation comes from—and where it doesn’t.

At 60 million rows, the fastest run finished in 15.5 minutes. The slowest took 66 minutes. Both ran on the same two cores, through the same platform. The difference comes down to pipeline type and how much output each one writes—not the engine itself. Pipelines that aggregate data down to a small result set finish fast. Pipelines that write millions of rows back to PostgreSQL take longer because the destination is the constraint, not the platform.

Where the Time Goes

Raw totals are useful. But understanding where the time goes tells you something deeper about how the platform is built—and whether it was built well.

Every run was tracked across four phases: Read, Transform, Write, and Overhead (session setup, query planning, orchestration coordination). The breakdown across scale tiers reveals a clear pattern:

At small volumes, session setup and query planning naturally dominate. There’s simply not much data to move, so initialization is the largest share of total time. But as volume increases, that overhead compresses dramatically. By 60 million rows, overhead has shrunk to just 4.5% of total time. The engine is spending over 92% of its runtime on actual data work: reading, transforming, and writing.

The absolute numbers behind that story are just as revealing. Overhead barely moves across the entire 1,000× range. Write time climbs the steepest at scale—and that’s a PostgreSQL insertion constraint, not a platform one.

The Bottleneck Tells the Story

At each scale factor, the dominant constraint shifted to a different layer. The progression tells you exactly what kind of system you’re looking at:

At no scale was DataFuseAI itself the limiting factor. As volume grew, the bottleneck moved naturally outward. First to the computational load of joins and aggregations. Then to the ingestion capacity of the PostgreSQL destination. The platform consistently processed its work ahead of what the external systems could absorb.

That is the right dynamic. A well-built pipeline platform should complete its own work faster than the systems it connects to can keep up. The bottleneck should always be somewhere else: in the database, in the network, in the compute layer. Never in the orchestration and transformation engine itself.

The 60 Million Row Runs, Up Close

The sf10 scale was the real test. Nearly 60 million rows pushed through joins, aggregations, filters, derived columns, and route-splits—on a machine with just two cores and 16 GB of memory.

The ranking by throughput shows a clear and consistent pattern. Aggregation pipelines—where output row counts are small—finish fastest because the write load is minimal. Write-heavy pipelines that push millions of output rows back into PostgreSQL take longer. The platform is not the variable. Output volume is.

The most write-intensive run was JOIN-SPLIT: joining 75 million rows and writing nearly 60 million output rows back to PostgreSQL. It completed in just over an hour on two cores. The volume of data being pushed into the destination database—not the pipeline platform—was the defining constraint.

Our Proof: Samples from the sf10 Runs

We don't show number, we present the proof.Below are direct screenshots taken from DataFuseAI during the sf10 benchmark runs—the same runs that produced the timing and throughput figures reported throughout this article. Each screenshot shows the completed pipeline, its recorded run time, and the status of the execution as logged by the platform.

These are unedited captures. No post-processing, no synthetic timing overlays. What you see is what the platform recorded.

AGG — lineitem sf10 Succeeded

JOIN-AGG — lineitem × orders sf10 Succeeded

FILTER-AGG — lineitem sf10 Succeeded

DERIVED — lineitem sf10 Succeeded

JOIN-SPLIT — lineitem × orders sf10 — Most Write-Intensive Run Succeeded

What This Means in Practice

Volume Handling, Confirmed

DataFuseAI completed complex ETL pipelines at 60 million rows on hardware well within the range of typical team environments. Throughput climbed from 1,571 rows per second at the smallest scale to a peak of 64,439 rows per second at the largest. That’s a 41× improvement, driven by the engine shifting from warm-up overhead into full productive work.

Economical Scaling

A 1,000× increase in data volume translated to just a 50× increase in processing time, on a two-core machine. Organizations growing their data operations do not need to proportionally scale their infrastructure to keep pace. The cost per unit of data drops sharply as volume rises—and then holds steady.

No Code Required

Every pipeline in this benchmark—joins, aggregations, derived columns, multi-sink routing—was built using DataFuseAI’s visual drag-and-drop interface. The same no-code environment a business analyst would use for a simple pipeline is the same one that handled 60 million rows of TPC-H data. There is no separate “power mode” or hidden scripting layer required to reach production scale.

What Wasn’t Optimized (And What That Means for You)

The benchmark was intentionally run under generalized, untuned conditions to establish an honest baseline:

• All PostgreSQL tables were unindexed, with no partitioning or query optimization applied
• Compute was the smallest practical setup: single node, 2 cores, 16 GB
• Batch sizes and parallelism were not tuned per pipeline or workload
• No PostgreSQL configuration tuning of any kind was applied

In a production deployment, each of these variables would be addressed. Indexed tables, multi-node clusters, tuned sink configurations, optimized batch sizing. Each of these would push performance further—some significantly.

Summary

31 pipelines. 4 scale factors. 8 pipeline types. A 1,000× range in data volume. Every second tracked and published.

DataFuseAI processed 60 million rows through complex ETL transformations—joins, aggregations, derived columns, multi-sink routing—on a 2-core, 16 GB machine.