Batch ETL processes data on a schedule — hourly, daily, or on demand — transforming and loading records in defined runs. Real-time streaming processes events as they arrive, with latency measured in seconds or less. The right architecture depends on what latency your downstream systems actually require, not on what latency is technically achievable.

The industry has spent the past several years framing streaming as the obvious upgrade path from batch. The framing is wrong for most pipelines. Streaming delivers lower latency; it also delivers exactly-once delivery semantics to manage, stateful processing to maintain, out-of-order events to handle, schema evolution to catch in live message streams, and consumer lag metrics where batch ETL had simple job pass/fail monitoring. Teams that choose streaming because it sounds more current — rather than because a downstream system requires sub-minute data freshness — inherit that operational complexity without a corresponding business return.

What follows covers what each architecture does mechanically, compares them across the dimensions that actually matter for selecting between them, identifies the specific downstream requirements that justify streaming's overhead, and provides a four-question framework for making the decision per pipeline rather than per organization.

What Each Architecture Actually Does

Batch ETL

Batch ETL extracts data from source systems at scheduled intervals, transforms it, and loads the results into a destination in bounded runs. Each run processes the data available since the previous run — either the full table or an incremental set identified by a timestamp watermark — and produces a discrete, auditable output. The run either succeeds and writes its output, or it fails and is retried. Monitoring is binary.

Batch runs can be scheduled at any frequency the business requires: hourly, every 15 minutes, twice daily, or weekly. "Batch" doesn't imply "slow" — it implies "scheduled." A pipeline running every 10 minutes is still batch architecture. The defining property is that processing happens in discrete, bounded runs with a defined start and end, not continuously.

Real-Time Streaming

Streaming processes events individually or in micro-windows as they arrive at the pipeline, producing outputs with sub-second to sub-minute latency. Events flow from producers (source databases, application event buses, IoT sensors, clickstreams) through a durable message log — Apache Kafka is the most common implementation — and are consumed by processing engines (Apache Flink, Spark Structured Streaming, Kafka Streams) that apply transformation logic continuously.

The defining properties of streaming distinguish it from fast batch in three ways. Processing is stateful: the pipeline must maintain state across events to compute aggregations, detect patterns, or join streams. Events may arrive out of order due to network delays, and the system must decide how to handle late data. Delivery guarantees — at-most-once, at-least-once, exactly-once — must be configured explicitly rather than assumed. Each property requires infrastructure, configuration, and operational expertise that batch ETL doesn't.

On micro-batch as a middle path: Micro-batch processing groups events into small time windows — typically one to five minutes — and processes each window using batch semantics. It delivers near-real-time freshness with simpler consistency guarantees than continuous streaming. For many use cases labeled as "real-time," micro-batch at two-to-five-minute windows satisfies the actual downstream requirement at meaningfully lower operational complexity than a continuous event-by-event processing architecture.

Latency, Cost, and Operational Reality: The Architecture Comparison

Batch ETL vs real-time streaming across the dimensions that drive architecture selection. The right choice depends on downstream latency requirements, team capability, and the specific failure modes each architecture introduces.
Dimension Batch ETL Real-Time Streaming
Data latency Polling interval (minutes to hours) Sub-second to sub-minute
Processing model Bounded runs — discrete start and end Continuous — events processed as they arrive
State management Stateless per run — no cross-run state required Stateful — aggregations and joins require fault-tolerant state stores
Delivery guarantee Implicit exactly-once — run succeeds or fails as a unit Must be configured explicitly: at-most-once / at-least-once / exactly-once
Out-of-order event handling Not applicable — all data is present at run time Requires watermarking strategy; late data may be dropped or reprocessed
Schema evolution Handled at pipeline deployment time, explicitly Schema changes arrive in live message streams; consumers must handle in real time
Failure monitoring Job pass / fail — binary, simple alerting Consumer lag metrics — continuous, requires lag threshold alerting
Infrastructure requirements Scheduler + compute + destination Message broker (Kafka) + stream processing engine + state backend + destination
Operational expertise required Pipeline scheduling, transformation logic, SQL Distributed systems, stream processing semantics, Kafka administration, state management
Best for Reporting, analytics, dashboards, reference data, regulated audit pipelines Fraud detection, real-time inventory, operational decisions with sub-minute SLAs

DataFuseAI supports batch ETL pipelines across 50+ connectors with scheduled execution, job failure monitoring, and three GA deployment models. Streaming (Kafka, Kinesis, Pub/Sub) is not currently supported.

Streaming is an SLA decision. The question isn't whether sub-second latency is technically achievable — it is. The question is whether any downstream system produces a materially different outcome when data is 30 seconds old versus 15 minutes old. Most don't.

When Real-Time Streaming Is Genuinely Required

Streaming earns its operational overhead in a specific set of downstream pipeline requirements. These aren't preferences — they're conditions where batch ETL's polling-interval latency produces materially worse business outcomes, where regulations explicitly require continuous monitoring, or where the event volume and ordering properties of the source make batch extraction architecturally impractical.

1. Fraud detection and real-time risk scoring. A payment fraud model that scores a transaction after it clears a nightly batch run isn't a fraud model — it's a loss reporting mechanism. Detection requires the transaction to be scored before authorization, or within seconds of event receipt. Financial institutions operating under SEC Rule 15c3-5[1] — the Market Access Rule — are required to maintain risk management controls that establish pre-trade limits on financial exposure and are implemented in real time, before orders are routed to an exchange. The requirement is explicit: controls must operate continuously, not at batch intervals. For broker-dealers, streaming isn't an architecture preference — it's a compliance requirement for this specific control class.

2. Real-time inventory allocation in high-velocity commerce. Consider a flash sale where 500 units of a product sell in four minutes. A batch ETL pipeline refreshing inventory counts hourly allows thousands of orders to commit against inventory that doesn't exist — each producing a downstream cancellation, a customer service interaction, and a reputational consequence. Streaming inventory pipelines update available counts as each order commits. The downstream decision — whether to allow the next order — requires data freshness measured in seconds. This is a condition where latency directly determines the correctness of an operational outcome, not the quality of a report.

3. Continuous sensor and telemetry monitoring with automated response. Manufacturing equipment emitting failure precursor signals, network infrastructure reporting anomalous packet patterns, or medical devices transmitting patient vital signs all share a common architectural requirement: the downstream system must detect specific event patterns and trigger a response within a time window where action remains possible. Processing equipment sensor data in hourly batch runs doesn't detect a bearing failure before it causes a line stoppage. The use case requires continuous event processing — not because streaming is elegant, but because the detection window closes within seconds of the signal appearing in the data.

4. GDPR-compliant deletion propagation at scale. GDPR Article 5(1)(e)[2] requires that personal data not be retained beyond its processing purpose. When a data subject exercises their right to erasure, organizations that replicate personal data across dozens of downstream systems face a propagation problem: how quickly does the deletion reach every system holding that data? Streaming pipelines propagate deletion events to downstream consumers in seconds. Batch ETL propagates them at the next polling interval — which, for daily pipelines, means personal data may persist in replicated systems for up to 24 hours after the erasure request was honored at the source. Whether that delay creates GDPR exposure depends on the specific processing purpose and the documented commitments in the relevant data processing agreements.

5. Event sources where data doesn't rest. Clickstream events, IoT telemetry at high frequency, financial market tick data, and real-time location feeds share a property: the meaningful unit of data is the event sequence, not a periodic snapshot. A daily extract of market tick data misses the intra-day price movements that constitute the actual signal. Batch ETL against these sources produces a compressed representation of what happened — useful for historical analysis, but not for systems that need to reason about the event sequence itself.

On the scope of streaming requirements: These five conditions are specific. They share a common property — the downstream system produces materially worse outcomes when data is minutes old rather than seconds old, and that worsening has a measurable business or regulatory consequence. Streaming architectures are correctly chosen when at least one of these conditions applies to the specific pipeline. Most pipelines in a typical data platform serve neither fraud detection nor real-time inventory allocation. Evaluate per pipeline, not per organization.

The Hidden Operational Costs of Streaming in Production

Streaming system documentation covers setup and configuration. Production operations reveal the costs. Five specific operational burdens distinguish streaming from batch ETL — not as theoretical risks, but as day-to-day operational realities for teams running continuous pipelines at scale.

Exactly-once delivery requires explicit implementation

Batch ETL achieves exactly-once semantics naturally: a run either completes and writes its output, or it fails and is retried without partial writes to the destination. The guarantee is structural. Streaming must implement it explicitly through idempotent consumers, distributed transactions, or consumer offset management combined with transactional sinks. At-least-once delivery — the default for most streaming configurations — means a consumer processes some events more than once under failure and recovery conditions. For pipelines where duplicate events produce incorrect financial totals, double-counted records, or erroneous audit entries, exactly-once is mandatory. It's also the most operationally demanding configuration to maintain in production, requiring careful coordination between the message broker, the processing engine, and the destination system.

Stateful processing introduces fault-tolerant state management

A batch ETL pipeline that computes a rolling seven-day average of order revenue reads all records in the window at run time and produces the result. Stateless. A streaming pipeline computing the same metric continuously must maintain the aggregation state across events — updating the running total as each new order arrives, expiring old values as they age out of the window. That state must be persisted to a fault-tolerant store so that a consumer restart doesn't lose the accumulated aggregation and produce an incorrect output. State backend selection, checkpoint interval configuration, and state store sizing add infrastructure and configuration surface area that batch pipelines don't require.

Out-of-order events require watermark strategy decisions

Network delays, retries, and distributed system timing mean events don't always arrive in the order they were produced. An order event timestamped at 14:32:00 may arrive at the stream processor after an event timestamped at 14:32:05. Windowed aggregations — computing metrics over time windows — must decide: how long to wait for late events before closing the window and producing the output? A five-second allowance for late arrival produces correct results when events are delayed by up to five seconds. Events arriving six seconds late are either dropped or trigger window recomputation, depending on the processing engine's configuration. Batch ETL doesn't face this decision — all data is present at run time, ordered by the query's sort clause. Streaming teams decide the watermark strategy at design time and live with its consequences when late events arrive at higher-than-expected frequencies.

Schema evolution arrives without deployment control

When a source application team adds a field to an event payload, batch ETL pipelines encounter the change at the next scheduled run — handled through explicit transformation logic updated at deployment. Streaming consumers encounter the schema change in the live message stream, potentially mid-run. Depending on serialization format (Avro with a schema registry handles this better than raw JSON), consumer behavior on encountering an unknown field varies from graceful schema evolution to consumer exception and lag accumulation. Teams running streaming pipelines against sources owned by other teams — common in microservices architectures — need schema governance processes to receive advance notice of schema changes. Without them, schema evolution in the source becomes an unscheduled incident in the stream consumer.

Consumer lag replaces job monitoring

Monitoring a batch ETL pipeline means checking whether the scheduled job succeeded — a binary outcome, alertable with a simple success/failure condition. Monitoring a streaming pipeline means tracking consumer lag: how far behind the consumer's current position is from the head of the message log. A consumer that is 10,000 events behind at a given moment may be recovering normally from a brief slowdown, or it may be falling progressively further behind and will never catch up without intervention. Distinguishing between acceptable lag variation and a genuine consumer failure requires lag rate-of-change monitoring, lag threshold alerting calibrated to the specific pipeline's throughput characteristics, and on-call procedures for streaming-specific incidents. Teams without distributed systems operational experience encounter this as a new monitoring discipline after deploying streaming — one that doesn't translate directly from the job pass/fail monitoring they had before.

When Batch ETL Is the Right Architecture

Batch ETL is the correct architecture for the majority of production data pipelines. Not because it's simpler — though it is — but because the pipelines most organizations run don't have downstream systems that produce materially worse outcomes when data is 15 minutes old versus 30 seconds old.

✓ Batch ETL Fits These
  • Business intelligence dashboards refreshed daily or hourly
  • Data warehouse loading from OLTP source systems
  • Financial reporting pipelines requiring reconcilable audit trails
  • Reference and dimension table refreshes (product catalogs, customer profiles)
  • Regulated data pipelines where compliance requires transformation-before-load
  • On-premise or air-gapped deployments with no streaming infrastructure
  • Append-only source tables (transaction logs, event history)
  • Analytics pipelines where source schema evolves frequently
✗ Batch ETL Doesn't Fit These
  • Fraud scoring before transaction authorization
  • Real-time inventory allocation preventing overselling at high velocity
  • Regulatory requirements for continuous pre-trade risk monitoring
  • Equipment failure detection with sub-minute response windows
  • Erasure propagation with strict sub-minute deletion SLAs
  • Event sequence analysis where intermediate states are the signal

Regulated data pipelines are a notable case where batch ETL isn't just acceptable — it's architecturally preferable. Pipelines processing personal data under GDPR or PHI under HIPAA often require transformation-before-load, meaning raw sensitive data must be cleaned, masked, or filtered before reaching the destination system. Batch ETL performs transformation in the pipeline layer before loading. Streaming architectures that feed data into a message bus typically move raw events — including unmasked personal data — before any transformation runs downstream. For pipelines where compliance requires that sensitive data never reach certain systems in its raw state, batch ETL is the architecturally correct choice regardless of latency preferences.

For the full treatment of where transformation timing intersects with compliance, ETL vs ELT: A Decision Framework for Pipeline Architecture covers when transformation-before-load is a legal requirement rather than an architectural preference.

DataFuseAI's batch ETL pipelines processed 60 million rows per pipeline run on a 2-core, 16 GB Databricks cluster — reaching a peak throughput of 64,439 rows per second — with no pipeline code written by the team running it. The operational model is scheduled execution, job failure history across the last five cycles, 10-second system metrics, and email alerts on scheduled job failure. For teams whose pipeline requirements sit in the left column above, that operational model covers what the pipeline actually needs. The pipeline automation solution page covers how the scheduling and monitoring model works in practice.

A Four-Question Decision Framework

Apply these questions in sequence, per pipeline. A clear answer at any step stops the analysis. The architecture follows the first question that produces a definitive constraint.

01

What latency does the downstream system actually require?

Name the specific downstream system and its SLA — not the desired freshness, the required freshness.

Every pipeline has a downstream consumer: a dashboard, a recommendation engine, a fraud model, a reporting database, an operational API. Each consumer has a data freshness SLA — either explicit (the fraud model must score within 500ms of event receipt) or implicit (the finance team reads the dashboard each morning). Write that SLA down. If the answer is "we'd prefer fresher data" rather than "the downstream system breaks or produces wrong outputs when data exceeds X minutes old," the latency requirement is implicit and flexible. Batch ETL at a frequency that meets the implicit preference is the correct architecture.

If the downstream SLA is sub-minute: Continue to Step 2. If the downstream SLA is minutes or longer: Batch ETL. Determine an appropriate polling frequency and stop here.

02

Does stale data produce measurably worse downstream outcomes?

Identify the specific decision the downstream system makes and whether latency changes that decision's outcome.

A fraud model operating on 15-minute-old transaction data scores a transaction that already cleared — detection is retrospective, not preventive. The outcome changes materially with latency: false negatives increase, fraud losses increase by a measurable amount per fraudulent transaction approved. That's a measurable business consequence of batch latency.

A sales performance dashboard used in weekly planning meetings doesn't produce materially different insights when data is 30 minutes old versus 30 seconds old. The downstream decision — resource allocation, quota planning, pipeline prioritization — operates at a weekly cadence. Data freshness at the minute level doesn't change the decision quality. There's no measurable business consequence of batch latency for this use case.

If stale data produces measurably worse outcomes with a quantifiable consequence: Continue to Step 3. If stale data doesn't change downstream decision quality: Batch ETL. Streaming adds operational complexity without business return for this pipeline.

03

Does the source generate events that can't be collected in periodic batches?

Assess whether the source data is event-native or batch-extractable.

High-frequency sensor telemetry, financial tick data, and real-time location feeds are event-native: the data is a continuous stream of timestamped events, and the meaningful signal is often in the event sequence rather than a periodic snapshot. Extracting these sources in hourly batches loses the intra-period pattern — the bearing failure signal that appeared and resolved between batch runs, the tick sequence showing a flash crash unfolding over 90 seconds.

Most OLTP databases, CRM systems, ERP platforms, and internal operational databases are batch-extractable: records are written transactionally and can be extracted on a schedule using incremental load patterns. The source doesn't mandate streaming architecture.

If the source is event-native and the event sequence is the signal: Streaming is architecturally required regardless of the downstream latency SLA. If the source is batch-extractable: Continue to Step 4.

04

Does your team have the operational capability to run streaming in production?

Assess honestly: Kafka administration, stream processing semantics, exactly-once configuration, consumer lag monitoring.

Streaming systems require operational skills that differ substantially from batch ETL: Kafka cluster administration, partition management, consumer group offset handling, stream processing engine configuration (Flink job graph management, Spark Structured Streaming checkpointing), state backend sizing, and lag monitoring procedures. A team choosing streaming architecture takes on those operational responsibilities regardless of whether it chose Kafka, a managed streaming service, or an embedded stream processing library.

This isn't a reason to avoid streaming when the downstream requirements genuinely justify it. A fraud detection pipeline at a payments company justifies building the operational capability. A retail analytics pipeline serving weekly planning meetings does not. Choose the architecture the downstream requirements require, then assess whether the team has or can acquire the operational capability to run it reliably. If the team can't — and the pipeline requirement genuinely requires streaming — the honest path is a managed streaming service that abstracts the Kafka administration layer, not batch ETL pretending to be streaming through a micro-batch with too-small windows.

On choosing streaming for future-proofing: "We might need real-time data eventually" is not a streaming requirement. It's an assumption about future requirements that may not materialize. Choosing streaming architecture today to avoid a migration later means paying streaming's operational costs now — for pipelines that don't need sub-minute latency — against a hypothetical future requirement. Build the architecture the current requirements justify. If requirements change, migrate then — with actual knowledge of what the new requirements are.

Common Questions About Batch vs Streaming

No. Streaming delivers lower latency at significantly higher operational cost: exactly-once delivery semantics, stateful processing, out-of-order event handling, and continuous consumer monitoring require expertise and infrastructure that batch ETL doesn't. For pipelines serving reporting, analytics, and most operational dashboards, batch ETL on an hourly or daily schedule meets every business requirement.

Streaming is the right architecture when downstream systems make materially different decisions based on seconds-old data versus minutes-old data — fraud detection, real-time inventory allocation, financial pre-trade risk controls that regulations require to operate continuously. Outside those conditions, streaming adds complexity without corresponding business value.

Micro-batch processing groups events into small time windows — typically one to five minutes — and processes each window as a mini-batch using batch semantics and simpler consistency guarantees. True real-time streaming processes events individually as they arrive, with sub-second to second-level latency and stateful, continuous processing.

For many pipelines labeled as "real-time," micro-batch at two-to-five-minute windows satisfies the actual downstream requirement with lower operational complexity than a continuous streaming architecture. If your downstream SLA is "data must be no more than five minutes old," micro-batch meets it. If the SLA is "data must be no more than five seconds old," continuous streaming is required.

Yes. Lambda architecture uses both patterns simultaneously: a streaming layer processes events continuously for low-latency results, while a batch layer reprocesses historical data periodically to produce authoritative, corrected outputs. Kappa architecture replaces the batch layer with a replayable streaming log, processing all data through the stream. Both are used by organizations with requirements that genuinely span both latency classes.

In practice, most organizations don't need a unified architecture. They run batch ETL for the majority of pipelines and a separate streaming system for the handful of use cases that require sub-minute freshness. The two operate independently. Adopting Lambda or Kappa architecture without that requirement adds coordination complexity between systems without the corresponding benefit.

Ask this specific question: if data is 15 minutes old instead of 30 seconds old, does the downstream system produce a materially different outcome with a measurable business consequence? For fraud detection, the answer is yes — a fraudulent transaction approved while waiting for the next batch run is a quantifiable loss. For a sales dashboard refreshed by the finance team each morning, the answer is no — 15-minute-old data and 30-second-old data produce identical business outcomes.

If the honest answer is no, batch ETL at an appropriate frequency meets your requirement at lower cost and complexity. If the answer is yes — and you can name the specific outcome that worsens and quantify the consequence — streaming's overhead is justified for that pipeline.

Exactly-once semantics guarantee that each event in a streaming pipeline is processed and written to the destination exactly one time — not dropped (at-most-once) and not written multiple times (at-least-once). Batch ETL achieves this structurally: a run completes and writes its output, or it fails and is retried without partial writes. Streaming must implement exactly-once guarantees explicitly through distributed transactions, idempotent consumers, and checkpointing mechanisms coordinated across the message broker, the processing engine, and the destination.

For pipelines where a duplicate event would cause an incorrect financial calculation, a double-counted record, or an erroneous audit entry, exactly-once is mandatory — and it's the most operationally demanding streaming configuration to maintain in production. At-least-once delivery (the streaming default) is acceptable only when consumers are idempotent: processing the same event twice produces the same result as processing it once.

The batch vs streaming decision follows downstream requirements. Streaming earns its five operational complexity layers — exactly-once delivery, stateful processing, watermarking, live schema evolution, and consumer lag monitoring — when downstream systems make decisions with measurable consequences that depend on sub-minute data freshness. For everything else, batch ETL at the right frequency delivers the same business outcome with binary monitoring, explicit schema change handling, and an operational model that doesn't require distributed systems expertise to run at two in the morning when something breaks.

For teams evaluating which pipelines in their stack genuinely require streaming versus which are good candidates for scheduled batch ETL, the CDC Explained piece covers the closely related question of when near-real-time extraction via log reading is necessary — another decision point that resolves to "less often than teams expect."