Integration Change Data Capture (External)
Problem
A business partner needs a feed of changes from one of your internal data stores, and the partner owns the receiving end: its API, message queue or SFTP server. Deliveries leave your private subnet through a perimeter egress gateway and authenticate to the partner with mTLS, OAuth 2.0 client credentials or SSH keys; the partner never gets a path into your database. Full-table exports or frequent polling load the source, risk its operational integrity and still leave the partner with stale data.
Solution
Use Change Data Capture (CDC) to pick up incremental changes in the workload's data store. Publish them asynchronously as events or streams, or batch them into files, and deliver them through a controlled egress point (an Egress Gateway or a secure data transfer service) to the partner's API or over a secure file transfer protocol.
Cloud Paradigm
- Event-Driven Architecture / Data Streaming
- Asynchronous Integration Patterns
- Secure Managed Data Transfer
- Data Observability
Solution Flow
Data Change Capture and Staging Flow:
- Backend Workload Data Store: An internal backend workload creates or changes data in its persistent data store (e.g., relational database, NoSQL data store).
- Change Data Capture (CDC) Component: A dedicated CDC component in the Private Subnet (Workloads) watches the store for changes (e.g., via transaction logs, triggers or logical replication).
- Data Transformation & Event Generation: It captures changes, optionally transforms them (e.g., filtering, light schema mapping), and publishes them as structured events (e.g., JSON, Avro) to a durable message broker or object storage bucket.
- Durable Event/Data Store: Changes are held in a highly available, scalable message broker (e.g., managed message queue, event streaming platform) or object storage (e.g., for batch file staging).
External Data Delivery Flow:
- Data Processing & Aggregation (Optional): For batch delivery, a processing workload reads events or files from the durable store, aggregates them and prepares the payload.
- Outbound Connection Initiation: The processing workload (or the CDC component itself, for real-time streaming) opens a secure connection to the external system.
- Egress Gateway / Secure Data Transfer Service: All outbound traffic passes through a Managed NAT / Egress Gateway or a Secure File Transfer Gateway in the Public Subnet (Perimeter), which enforces egress policies (allow-listed FQDNs/IPs), performs NAT and secures the transport.
- External Target System: The partner receives the data at its API endpoint (synchronous or asynchronous), at its message queue, or over a secure file transfer protocol (e.g., SFTP, HTTPS).
When to Use
- You must send incremental changes to an external partner or system without giving it any access to your source data store.
- The source has strict operational integrity requirements, and log-based CDC captures changes without taxing production transactions.
- You need near-real-time or micro-batch delivery, and full periodic dumps would be too slow, too heavy or too stale.
- Outbound data must pass controlled, auditable egress points with FQDN/IP allow-listing and secure transport.
- Both streamed events and aggregated files are viable, and you want one capture mechanism to feed either delivery mode.
When NOT to Use
- The external system needs synchronous request/response queries on current state. Expose a purpose-built API instead.
- You need a one-off or infrequent bulk migration. A plain batch export avoids standing up CDC infrastructure.
- The consumer must write back to the source, which needs a full data synchronisation or master-data pattern.
- Heavy transformation, enrichment or multi-source joins are central to the flow. Hand them to a dedicated integration or ETL platform.
- Your data store has no workable change tracking (no transaction log access, triggers or reliable change columns).
Trade-offs
- Minimal source impact from log-based capture vs running and monitoring a separate CDC component and durable broker.
- Decoupled, resilient asynchronous delivery vs eventual-consistency lag between a source change and the partner receiving it.
- Strong security behind an egress gateway vs extra perimeter complexity and FQDN/IP allow-list upkeep.
- At-least-once durability with retries and DLQs vs idempotency and de-duplication logic at the external consumer.
- Flexible streaming or batch payloads vs the overhead of versioned schema governance and format management.
Illustrative Scenario
A hypothetical scenario showing how the pattern applies. Not a client case study.
In this hypothetical case, an automotive manufacturer streams production and inventory updates to its franchised dealer network. A log-based CDC component in the private workloads subnet tails the transaction log of the vehicle-order database. It captures each build-status change, VIN allocation and pricing adjustment, and publishes them as versioned Avro events to a managed event stream for dealers that want live updates. Some dealers prefer nightly reconciliation to a live stream. For them, a batch processing workload collects the day's events into Parquet files, records the last committed sequence number for idempotency, and sends the files out through a Secure File Transfer Gateway in the perimeter subnet. That gateway enforces FQDN allow-listing and NAT. Each dealer management system receives events at its own API endpoint, or files over SFTP, and never touches the manufacturer's source database. Dead-letter queues, retry logic and transfer metrics keep delivery observable and recoverable across a network of thousands of dealerships.
Additional Details
- CDC Implementation: Choose log-based CDC, trigger-based CDC or periodic polling with change detection, to suit the source data store and the latency you need.
- Event/File Format: Define one consistent, versioned format for published changes (e.g., JSON, Avro, Parquet, CSV).
- Error Handling & Idempotency: Use dead-letter queues for events, retries, and idempotency at the external consumer, so duplicate deliveries do no harm. Record the last successful publication timestamp or sequence number.
- Scalability & Resiliency: Make the CDC component and the message broker or object storage highly available and able to scale with data volume.
- Observability: Log, monitor and alert from the CDC component to the external delivery endpoint. Track data volume, latency, successful transfers and errors.
- Direct Database Access Prohibition: Never allow external parties to connect directly to the database. This pattern keeps an abstraction layer in between.
- Pattern Delegation: Give complex transformation or routing logic to a dedicated data integration platform or event processing workload, not to the CDC component.
Security Controls
- Perimeter Security: All outbound data starts in a Private Subnet (Workloads) and passes through a dedicated Egress Gateway or Managed NAT in a Public Subnet (Perimeter), giving one point of control and logging.
- Transport Security: Enforce Transport Layer Security (TLS) 1.2 or higher from the CDC component to the message broker or object storage, and from the egress point to the external target system.
- Authentication & Authorisation:
- For API-based exchange, use OAuth 2.0 (Client Credentials Grant) or Mutual TLS (mTLS) for system-to-system authentication.
- For secure file transfer, use SFTP with SSH key-based authentication or HTTPS with client certificates.
- Keep all credentials (API keys, certificates, SSH keys) in a secrets management service, never in cleartext.
- Data Integrity & Immutability: Protect integrity through capture and transmission. For object storage, consider immutability and versioning.
- Outbound Controls: Allow egress only to explicitly allow-listed external IP addresses, fully qualified domain names (FQDNs) or predefined endpoints. Apply Data Loss Prevention (DLP) measures if sensitive data is involved.
- Data Masking/Anonymisation: Mask, anonymise or tokenise sensitive or personally identifiable information (PII) before it leaves the internal environment.