Change data capture (CDC) reads a database's transaction log to record only the rows that changed since the last extraction — inserts, updates, and deletes — rather than querying source tables on a schedule. It produces a stream of per-operation events rather than periodic snapshots. Whether that capability justifies the operational complexity depends on four specific pipeline conditions.
CDC appears in a lot of vendor marketing and conference talks. The technology is genuinely useful for a narrow set of requirements: tracking hard deletes, feeding operational systems that make real-time decisions, reading from databases that can't absorb query load, and building audit trails that capture every intermediate row state. Outside those requirements, CDC adds schema evolution management, log retention dependencies, consumer lag monitoring, and ordering complexity that incremental batch ETL handles more simply and with fewer failure modes.
What follows covers what CDC is mechanically, how it compares architecturally to batch ETL and incremental load, the four specific conditions where CDC's overhead is justified, and the production problems that surface when CDC is chosen by default rather than by requirement.
What Is Change Data Capture (and How Does It Work)?
Change data capture is a technique for extracting data changes from a source system by reading the database's internal transaction log — the ordered record the database itself maintains of every committed write — rather than issuing queries against source tables. The result is a change stream: an ordered sequence of row-level operations, each carrying a timestamp and the before/after state of the row.
Every major relational database maintains a transaction log. PostgreSQL calls it the Write-Ahead Log (WAL). MySQL and MariaDB expose it as the binary log (binlog). Oracle provides redo log access via LogMiner. SQL Server has native Change Tracking and Change Data Capture engine features. CDC tools read that log and translate each committed event into a structured record that downstream consumers can process.
Three mechanisms implement this:
Log-based CDC reads the database's native transaction log directly. It adds no load to the source system's query executor — the log read is external to the query path — and requires no schema modifications. This is what most practitioners mean when they say "CDC" in 2026.
Trigger-based CDC installs database triggers that write change events to a shadow audit table on inserts, updates, or deletes. It requires schema modifications to the source database and adds write overhead proportional to change volume. Less common in new implementations for this reason.
Query-based polling uses scheduled queries against source tables, comparing an updated_at column to the last extraction time. This is incremental load, not CDC. The distinction matters — and is the subject of the next section.
The defining architectural property of log-based CDC: it records one event per database operation, in the order those operations were committed. A row updated five times between two polling intervals produces five events, each with the intermediate state. Batch ETL with timestamp watermarking produces one record per row — the final state at polling time — with no record of what happened in between. That difference determines when CDC is necessary and when it isn't.
CDC vs Batch ETL vs Incremental Load: The Architecture Comparison
Three extraction patterns exist for moving data from source systems to destinations. They differ in how changes are detected, what load they place on the source, and what guarantees they provide about the state of data at the destination.
| Dimension | Full Extract | Incremental Load (watermark) | CDC — Log-based |
|---|---|---|---|
| How changes are detected | Full table scan every run | Rows where updated_at exceeds last run time |
Database transaction log events |
| Load on source system | High — reads all rows on every run | Medium — reads changed rows via query | Low — reads log outside the query path |
| Data latency | Polling interval | Polling interval | Near-real-time (seconds to minutes) |
| Captures hard deletes | No — deleted rows simply disappear | No — deleted rows disappear from queries | Yes — DELETE events recorded in log |
| Captures intermediate states | No — final state at run time only | No — final state at poll time only | Yes — every committed operation recorded |
| Schema change handling | At transformation time, explicitly | At transformation time, explicitly | In the change stream — requires consumer-side handling |
| Operational complexity | Low | Low to moderate | High |
| Best for | Small tables, reference data, full refreshes | Large tables, slow-changing data, append-heavy sources | Real-time operational systems, delete tracking, intermediate-state audit trails |
The table clarifies a common confusion. Incremental load with an updated_at watermark is not CDC. It is a query-based extraction that handles most common "only changed rows" requirements — at the polling granularity — without the operational complexity of log reading. It covers updated rows. It does not cover deleted rows, and it shows only the final state of a row between polling intervals, not every operation that occurred.
CDC adds exactly two capabilities that incremental load cannot provide: delete capture and per-operation event history. Whether either capability is genuinely required for a specific pipeline is the correct question to ask before choosing the architecture. It's also the question that gets skipped most often.
For a deeper treatment of the ETL vs ELT decision that underlies where transformation happens in these patterns, ETL vs ELT: A Decision Framework for Pipeline Architecture covers the transformation timing tradeoffs in detail.
When CDC Is Genuinely the Right Choice
CDC's operational overhead is justified when at least one of the following is true for the specific pipeline. These aren't guidelines — they're the specific conditions CDC was designed to handle and that batch ETL patterns cannot satisfy.
1. Hard deletes must be tracked downstream. When records are physically removed from the source table rather than flagged with a soft-delete marker, batch ETL and incremental load lose those records silently — a deleted row simply stops appearing in extraction queries. Downstream systems that need to know a record no longer exists (account closures, GDPR erasure requests, inventory removals) require either CDC or a source schema change to add a soft-delete mechanism. When source schema modifications aren't possible, CDC is the architectural answer. This is the most common legitimate CDC requirement across production data stacks.
2. Downstream decisions require sub-minute data freshness. Fraud detection systems scoring transactions as they arrive, real-time inventory allocation preventing overselling, and operational state machines that must reflect the current state of a record within seconds benefit from CDC's near-real-time event stream. The diagnostic question: does the downstream system produce materially different outcomes depending on whether data is 30 seconds old versus 15 minutes old? For most reporting and analytics use cases, the answer is no. For operational pipelines where decisions have immediate financial or user-facing consequences, the answer may be yes.
3. Source databases cannot absorb query load. Performance-sensitive production databases — high-traffic OLTP systems, financial transaction databases during peak hours, medical record systems with strict latency SLAs — sometimes can't accommodate the additional query load that even incremental scans create. Log-based CDC reads the transaction log without issuing queries against source tables. It operates outside the query execution path entirely, adding effectively zero contention to application transactions. When the DBA's response to "can we run hourly extracts against this table?" is "absolutely not," CDC is the extraction mechanism that doesn't compete with application traffic.
4. Compliance or audit requirements need every intermediate state. Some regulatory and audit requirements need a record of every state a row passed through, not just its state at extraction time. A financial record updated three times between hourly polling intervals has three intermediate states that batch ETL never captures. HIPAA's audit control standard (45 C.F.R. § 164.312(b))[2] requires covered entities to implement mechanisms to record and examine activity in systems containing electronic protected health information — at the operation level, not just the snapshot level. CDC provides that record. Batch ETL, which captures the row's state at polling time, does not satisfy this requirement for systems where every data access and modification must be logged.
On soft deletes as a CDC alternative: Before implementing CDC solely to track deletes, check whether the source database can be modified to support soft deletion — flagging records with a deleted_at timestamp and a is_deleted boolean rather than removing rows. Incremental load with watermarking handles soft-deleted rows naturally, and the source change is usually a one-time schema modification. CDC is the right solution when that modification isn't possible.
The Hidden Operational Costs of CDC in Production
CDC documentation covers setup. Production operations reveal a different set of problems — four failure modes that batch ETL pipelines avoid entirely and that teams discover after deployment rather than before.
Schema evolution is the most common production problem. When a source database team adds a column, renames a field, or changes a data type, CDC consumers must handle that schema change in the live event stream. Depending on the CDC tool, the destination, and the change type, this can produce consumer failures, silent data truncation, or schema drift that propagates downstream before anyone notices. Batch ETL pipelines handle schema changes at the transformation layer — explicitly, at deployment time, with control over what the destination receives. CDC consumers face schema changes in real-time, without advance notice from the source team, and without the option to pause the stream while the mapping is updated. In environments where source schemas change regularly — development databases, rapidly evolving SaaS integrations — this becomes continuous maintenance rather than an occasional fix.
Transaction log retention creates resync requirements. Database transaction logs are not infinite. Retention windows vary from hours to days depending on database configuration and storage policy. A CDC consumer that falls behind the log head — due to a downstream system failure, a consumer crash, or a network partition — risks falling outside the retention window. When the consumer position is behind the oldest available log entry, the only recovery path is a full table resync: re-snapshotting the entire source table and resuming from that baseline. Teams that adopt CDC specifically to avoid full extracts still need to plan for full resyncs as a failure recovery procedure. Consider a production MySQL database with a 48-hour binlog retention period. A two-day maintenance window on the CDC consumer produces a scenario where recovery requires a full extract regardless. The "incremental" property of CDC is conditional on the consumer staying current.
Most teams don't need CDC. They need incremental ETL with proper output validation and failure alerting. CDC solves a specific problem. The operational cost arrives whether you needed that solution or not.
Ordering guarantees are not universal. Distributed CDC implementations where events flow through a message bus before reaching the destination introduce the possibility of out-of-order delivery. An update followed by a delete can arrive at the consumer in the wrong sequence under load, depending on partitioning configuration. Batch ETL avoids this entirely: each run produces a consistent snapshot of source state at a specific point in time. For pipelines where event ordering matters — financial transactions, state machine transitions — CDC requires explicit ordering mechanisms that add configuration and monitoring surface area.
Consumer lag requires a different operational model. Monitoring a batch ETL pipeline means checking whether the scheduled job succeeded or failed — a binary outcome that surfaces clearly in any job monitoring system. Monitoring a CDC pipeline means tracking consumer lag: how far behind the consumer position is from the current head of the transaction log. Teams without streaming operations experience encounter consumer lag as a new operational concept after deploying CDC — a metric that doesn't exist in batch pipelines and that requires different alerting thresholds, different SLA definitions, and different incident response procedures than job success/failure monitoring.
On CDC operational readiness: Before deploying CDC to production, define: (1) what consumer lag threshold triggers an alert, (2) what the full resync procedure is when the consumer falls outside log retention, (3) how schema changes in the source database are detected and communicated before they break consumers, and (4) who owns consumer lag monitoring. Teams that deploy CDC without answers to these questions discover them during incidents rather than during planning.
When Batch ETL or Incremental Load Is Enough
Incremental batch ETL covers the majority of production data pipeline requirements — with lower operational complexity, explicit schema change handling, and binary success/failure monitoring that surfaces problems immediately. The use cases where CDC provides no material benefit are broader than most teams expect.
Daily and hourly reporting and analytics. A pipeline refreshing a business intelligence dashboard on a daily or hourly schedule gets no useful improvement from sub-minute CDC latency. The stakeholders consuming the dashboard can't process data faster than the reporting schedule anyway. Incremental ETL delivers the same result with a simpler operational footprint and no consumer lag to monitor.
Dimension tables and reference data. Product catalogs, customer profiles, geographic reference data, and configuration tables change slowly and irregularly. Scheduling incremental extraction at whatever frequency the business requires — daily, every four hours — captures all changes without the overhead of continuous log reading. CDC is not proportionate to the change rate of slowly evolving data.
Append-only source tables. Event logs, transaction records, clickstream tables, and audit tables where records are only inserted and never updated or deleted are natural incremental load targets. Extract all records where created_at exceeds the last run timestamp. CDC adds no capability here — there are no updates to capture and no deletes to track.
Source systems that support soft deletes. When source databases mark deleted records with a timestamp flag rather than physically removing rows, incremental load picks up deletions automatically — the deleted_at timestamp is newer than the last extraction time. This covers the most common CDC justification without requiring log access.
Pipelines where source schemas are actively evolving. When a source system is under active development and the schema changes frequently, CDC's schema evolution problem becomes a continuous maintenance obligation. Batch ETL pipelines apply explicit transformation logic at the extract stage, handling schema changes at pipeline deployment time with full control over what reaches the destination.
The practical diagnostic question is not "do we want real-time data?" Nearly every team would prefer fresher data. The question is whether downstream systems make materially different decisions with 30-second-old data versus 15-minute-old data. For reporting, historical analytics, and most operational dashboards, the answer is no — and the correct architecture is incremental batch ETL with reliable scheduling and output validation.
For teams running database-centric pipelines against PostgreSQL, MySQL, Oracle, SQL Server, MongoDB, or AWS RDS variants, DataFuseAI's pipeline automation covers scheduled batch ETL with 50+ source connectors, monitoring across the last five execution cycles, and email alerts on job failure — the operational model most pipelines actually need. For a broader view of the connector landscape, the data connectors overview covers supported database types in detail.
CDC and Compliance: Where the Requirements Actually Intersect
CDC appears in compliance discussions regularly, but the relationship between CDC and specific regulatory requirements is more precise than "use CDC for compliance." Three distinct intersections are worth understanding.
GDPR data accuracy and timely deletion. GDPR Article 5(1)(d)[1] requires that personal data be "accurate and, where necessary, kept up to date." For organizations that replicate personal data across systems, CDC supports this principle by propagating updates in near-real-time rather than at the next batch polling interval. Article 5(1)(e) adds a storage limitation requirement — personal data must not be kept longer than necessary for its processing purpose. A CDC pipeline that propagates GDPR erasure requests to downstream systems within seconds can reduce the window during which deleted personal data persists in replicated systems. Batch ETL achieves the same outcome with a deletion propagation delay equal to the polling interval. Whether that delay creates compliance exposure depends on the specific processing purpose and the documented expectations in the relevant data processing agreement.
HIPAA audit trail completeness. HIPAA's audit control standard (45 C.F.R. § 164.312(b))[2] requires covered entities to implement hardware, software, and procedural mechanisms to record and examine activity in systems containing electronic protected health information. CDC's operation-level event log — recording each insert, update, and delete against PHI-containing tables as a discrete, timestamped event — directly satisfies this requirement for database activity. Batch ETL captures final row states at polling intervals. For PHI systems where the audit trail must capture every access and modification event, not just snapshots, CDC is the architectural pattern the regulation's technical requirements point toward. For a more detailed treatment of what audit-ready data infrastructure looks like in practice, Audit-Ready Data Operations: Build It In, Don't Bolt It On covers execution logging, access controls, and what compliance teams actually examine when auditing data infrastructure.
SOX data traceability. SOX Section 404 requires assessment of internal controls over financial reporting. Data pipelines feeding financial reporting systems need traceable records demonstrating that source data arrived at the destination accurately and completely. CDC provides a verifiable, timestamped record that every committed database operation was captured and delivered — a complete chain of custody from source to destination. Batch ETL with explicit row count reconciliation and checksum validation achieves equivalent traceability for most financial reporting pipelines. The choice between them depends on whether the specific internal control requires operation-level event history or whether end-state reconciliation satisfies the control objective.
NIST SP 800-53 Rev. 5[3] defines the AU (Audit and Accountability) control family — including AU-2 (Event Logging) and AU-12 (Audit Record Generation) — as the reference standard for U.S. federal system controls and the basis for most SOX IT audit frameworks. Both CDC and batch ETL pipelines can satisfy AU control requirements; the specific control objective determines which approach is technically appropriate.
CDC doesn't create compliance requirements, and batch ETL doesn't violate them by default. The architecture decision is driven by what a specific regulation's technical requirements ask for, applied to the specific data category and pipeline. When the requirement is per-operation event capture, CDC is the technical answer. When the requirement is end-state accuracy and reconcilable traceability, incremental batch ETL satisfies it. Treat compliance requirements as technical specifications, not as implicit endorsements of any particular architecture.
Common Questions About Change Data Capture
CDC does not replace ETL. It is one of several extraction patterns within a broader ETL or ELT architecture — CDC addresses how data is extracted, not how it's transformed or loaded. The transformation and load phases apply regardless of whether extraction uses full scans, incremental watermarking, or log-based CDC.
Most production data stacks run both patterns: CDC for the handful of pipelines that genuinely require near-real-time event capture or delete tracking, batch ETL for everything else. The decision is made per pipeline based on requirements, not per organization based on preference.
Incremental load queries source tables for rows where the modification timestamp exceeds the last extraction time. CDC reads the database's transaction log directly, without issuing queries against source tables. Three specific differences matter in practice.
Incremental load misses deleted rows — physically removed records disappear from queries without a trace. CDC captures DELETE events explicitly. Incremental load shows only the final row state between polling intervals. CDC captures every intermediate state as a discrete event. CDC also places no load on the source system's query executor, while incremental scans compete with application queries on the same database engine. For most pipelines where soft deletes are possible and hourly freshness is acceptable, incremental load is the operationally simpler choice.
Most major relational databases provide native log access for CDC. PostgreSQL uses Write-Ahead Log (WAL) logical replication, configurable via the wal_level parameter. MySQL and MariaDB expose the binary log (binlog), which must be enabled and configured for row-level format. Oracle provides redo log access via LogMiner. SQL Server has built-in Change Tracking and Change Data Capture engine features available in Enterprise and Standard editions. MongoDB provides change streams for document-level event capture.
Support varies by version and edition — some CDC features require specific editions or minimum versions. Verify your exact database version and edition against the CDC tool's compatibility matrix before planning an implementation. Log retention defaults also vary significantly across databases and may need adjustment for CDC consumers to function reliably.
Log-based CDC reads the database's native transaction log (WAL, binlog, redo log) to extract change events. It adds no load to the source database's query executor, requires no schema modifications to source tables, and captures events that the database would have recorded regardless of whether CDC was running.
Trigger-based CDC installs database triggers that fire on insert, update, or delete operations and write change events to a shadow audit table. It requires modifying source database schemas, adds write overhead proportional to change volume (each data write now produces two writes — the original and the trigger write), and creates performance implications on high-write tables. Log-based CDC has become the standard approach for production implementations because it doesn't affect application performance or require source schema changes.
CDC is not required by GDPR. The regulation establishes data quality and minimization principles (Article 5)[1] that can be satisfied by multiple architectural approaches. CDC supports specific GDPR obligations in some architectures — near-real-time propagation of erasure requests limits the window during which deleted personal data persists in replicated systems — but batch ETL achieves the same outcome at a polling-interval delay.
Whether that delay creates compliance exposure depends on the specific processing purpose, the data subject's reasonable expectations, and the documented terms in your data processing agreements. GDPR doesn't specify extraction patterns — it specifies data quality and protection outcomes. Verify your specific compliance requirements with qualified legal counsel rather than treating any architecture as a compliance default.
Most data pipelines don't need CDC. The majority serve reporting and analytics use cases where daily or hourly batch ETL meets every business requirement at materially lower operational complexity. CDC earns its overhead in four specific situations: hard-delete tracking, sub-minute latency requirements, source query restrictions, and per-operation audit trails. Outside those conditions, an incremental ETL pipeline with robust output validation, scheduled job monitoring, and failure alerting delivers equivalent outcomes — with fewer production failure modes, simpler schema evolution handling, and operational metrics any data engineer can monitor on a Monday morning without understanding consumer lag curves.
