Integration Change Data Capture (Internal)
Problem
One internal team needs the changes from another team's database, to keep its own store in sync, drive event processing or feed analytics. Both ends run in the same organisation's private subnets, so traffic stays on the private network and access rests on workload identities (managed identities or service accounts), IAM policies and network segmentation. The changes have to flow quickly while the source system's performance, schema and application code stay untouched.
Solution
Run a dedicated Change Data Capture (CDC) service that reads inserts, updates and deletes from the source workload's data store, converts each into a standard event format, and publishes it to a highly available, scalable event streaming platform or secure object storage. Subscribing workloads and data pipelines consume the events asynchronously.
Cloud Paradigm
- Event-Driven Architecture (EDA)
- Data Streaming and Event Sourcing
- Loosely Coupled Integration
- Managed Data Services
- Data Pipeline Automation
- Data Mesh Principles (for decentralised data ownership and consumption)
Solution Flow
Change Data Capture (CDC) Flow:
- Source Backend Workload: A backend workload performs transactional operations (e.g., insert, update, delete) on its persistent data store.
- CDC Service Integration: A dedicated CDC service in a private subnet (workloads) opens a secure, non-invasive connection to the source data store, using database transaction logs, triggers or vendor-provided CDC interfaces.
- Event Capture & Transformation: The service reads the transaction log or change streams and converts raw changes into a standard, structured event format (e.g., JSON, Avro, Protobuf), often with metadata such as operation type, timestamp and transaction ID.
- Event Publication: The events go to a highly available, durable event streaming platform (e.g., Kafka-compatible service) or secure object storage, which acts as the central record of changes.
Data Consumption Flow:
- Subscribing Backend Workload: A consumer workload or data pipeline in a private subnet (workloads) subscribes to the relevant topic(s) or bucket(s).
- Event Ingestion: The subscriber ingests the change events, usually in chronological order.
- Data Processing & Synchronisation: The consumer updates its own data store, triggers downstream business logic or feeds an analytical data lake, staying consistent with the source without direct coupling.
When to Use
- A consumer needs near-real-time incremental changes from another service's data store, and polling or nightly batch extracts fall short.
- The source system's performance and schema must stay untouched, which rules out application-level dual writes or trigger-heavy designs.
- Several independent consumers (sync targets, event processors, analytical lakes) must react to the same changes from one capture point.
- You need an ordered, durable, replayable history of changes, not just the current state of records.
- Teams are moving to event-driven architecture, but the source system is database-centric and cannot emit domain events.
When NOT to Use
- The source workload already publishes well-defined domain events. Consume those instead of reverse-engineering intent from row changes.
- Consumers only need periodic full snapshots or aggregate reports, where a scheduled batch ETL job is simpler and cheaper.
- You need synchronous request-response access to current data. Use an API or query interface instead.
- The source store has no reliable transaction log, change stream or CDC interface, which would force brittle trigger-based capture.
- The consumer needs strict cross-entity transactional consistency. CDC delivers eventual, per-row consistency.
Trade-offs
- Non-invasive, low-impact capture from transaction logs vs running and monitoring a dedicated CDC service and streaming platform.
- Loose coupling and many independent consumers vs eventual consistency, with every consumer built for idempotency and replay.
- Ordered, durable, replayable change history vs storage cost and retention-policy management on the streaming layer or object store.
- Standardised, schema-managed events vs the discipline of schema evolution, registries, and backward and forward compatibility.
- Scalable, resilient asynchronous processing vs extra end-to-end latency and the effort of tracking consumer lag, DLQs and pipeline observability.
Illustrative Scenario
A hypothetical scenario showing how the pattern applies. Not a client case study.
Take a hypothetical retail bank whose core deposit-accounting platform writes every transaction posting, balance adjustment and account status change to a relational database that must not be slowed or altered. A CDC service in a private workload subnet attaches to the database transaction log. It captures each insert, update and delete without touching the application, and turns it into an Avro event carrying the operation type, timestamp and transaction ID. The events go to a Kafka-compatible topic. A real-time fraud-scoring engine and an analytical data lake each subscribe to the topic independently and ingest the events in chronological order. Each consumer applies idempotent upserts, so it stays eventually consistent with the source even when events are redelivered. A schema registry keeps schema changes backward-compatible, and a dead-letter queue holds malformed records for reprocessing. Dashboards track consumer lag and end-to-end latency across the whole pipeline, and neither consumer ever queries the core database directly.
Additional Details
- Idempotency: Make consumers idempotent, so duplicate events or reprocessing after failures or scaling events on the streaming platform do no harm.
- Data Format & Schema Evolution: Standardise the event format (e.g., JSON, Avro) and use a schema registry to keep schema changes backward and forward compatible.
- Error Handling & Dead-Letter Queues: Handle errors in both the CDC service and the consumers. Send events that fail processing to dead-letter queues (DLQs) on the streaming platform for inspection and reprocessing.
- Observability: Monitor, log and alert on the whole CDC pipeline, tracking:
- Latency from source change to event publication and consumption.
- Throughput (events or messages per second).
- Error rates in capture, transformation and publication.
- Consumer lag (the delay between an event being published and processed).
- Resource use of the CDC service and event streaming platform.
- Scalability & Resilience: Make the CDC service and streaming platform highly available and able to absorb changing data volumes and source load. Use managed cloud services for event streaming where possible.
- Data Retention: Set clear retention policies for the streaming platform or object storage to control storage cost and meet compliance requirements.
Security Controls
- Transport Security: Enforce Transport Layer Security (TLS) 1.2 or higher between the CDC component, source data store, event streaming platform and consuming workloads.
- Data Encryption: Encrypt data at rest in the source data store, the event streaming platform and any transient storage the CDC service uses.
- Authentication & Authorisation:
- The CDC service authenticates to the source with least-privilege service accounts, managed identities or short-lived credentials.
- Identity and Access Management (IAM) policies let only authorised subscribing workloads read the change events in the streaming platform or object storage.
- Secure Credential Management: Keep credentials (e.g., database connection strings, API keys) in a managed secrets management service, not in application code or configuration files, and rotate them regularly.
- Network Segmentation: Place the CDC service, source data store and streaming platform in private subnets (workloads) with strict network access controls (e.g., Security Groups, Network ACLs) that limit ingress and egress to required endpoints.
- Data Governance & Masking: Mask or tokenise sensitive data elements in the change events before publishing, where compliance or privacy rules require it.
- Audit Logging: Log all CDC operations, including captures, publications and access attempts, for audit and compliance.