Integration Event Streaming (Cloud)
Problem
Two external systems or organizations must exchange data in real time, but direct peer-to-peer integration is impractical or undesirable. Without an intermediated streaming layer, brittle point-to-point links create tight coupling and unreliable delivery, leaving events lost, delayed, or duplicated across organizational boundaries.
Solution
Establish a fully managed, cloud-native Event Streaming Platform to facilitate asynchronous, real-time data exchange between external producers and consumers. External producers publish event streams to designated topics, while external consumers subscribe to relevant topics or queues to process events. Access to the platform components from external systems is secured and mediated via a Web Application Firewall (WAF) or dedicated network connectivity within a Public Subnet (Perimeter).
Cloud Paradigm
- Event-Driven Architecture (EDA)
- Asynchronous Messaging
- Hybrid Integration
- Cloud-Native Streaming
- Serverless Event Processing
Solution Flow
Data Ingestion Flow (External Producer to ESP):
- External Producer: An external application or system generates events and initiates a secure connection (e.g., HTTPS, Kafka protocol over TLS) to the Event Streaming Platform.
- Web Application Firewall (WAF) / Dedicated Network: Traffic from the external producer traverses the Public Internet to a Web Application Firewall (WAF) in the Public Subnet (Perimeter) which inspects requests, or it flows over a dedicated private network connection.
- Event Streaming Platform (ESP) Ingress: The ESP's ingestion endpoint receives the authenticated and authorized events. The platform then publishes these events to the designated topic(s) or stream(s).
- Data Persistence: Events are durably stored within the ESP, awaiting consumption.
Data Consumption Flow (ESP to External Consumer):
- External Consumer: An external application or system establishes a secure connection to the Event Streaming Platform to consume events.
- Web Application Firewall (WAF) / Dedicated Network: The connection request from the external consumer similarly traverses the Public Internet via a Web Application Firewall (WAF) or a dedicated private network.
- Event Streaming Platform (ESP) Egress: The ESP's consumption endpoint (e.g., queue, partition group) delivers events to the authenticated and authorized external consumer, respecting consumption offsets and acknowledgments.
- External System Processing: The external consumer processes the received events according to its business logic.
When to Use
- When two or more independent organizations need real-time, event-driven data exchange but cannot expose internal systems for direct peer-to-peer coupling.
- When producers and consumers must scale, deploy, and fail independently, with the platform absorbing load spikes and consumer downtime via durable storage.
- When you need a neutral, intermediated boundary that centralizes authentication, authorization, and traffic inspection for all external parties.
- When multiple external consumers must independently subscribe to the same event streams at their own pace, using consumption offsets.
- When schema evolution across organizational boundaries must be governed through a shared registry.
When NOT to Use
- When integration is a simple, synchronous request/response needing an immediate reply — an API gateway or REST facade fits better.
- When only two internal systems within the same trust domain exchange data; a lighter internal message broker avoids perimeter overhead.
- When data volumes are low and infrequent, where batch file transfer or scheduled ETL is cheaper and simpler.
- When strict transactional consistency across systems is required, since event streaming favors eventual consistency.
- When one party mandates a direct private connection contractually and no intermediary is permitted.
Trade-offs
- Loose coupling and independent scaling of external parties vs the operational cost of running and securing a highly available managed streaming platform.
- Durable buffering that tolerates consumer outages vs eventual-consistency semantics and the need to handle offsets, replays, and duplicates.
- Centralized, inspectable perimeter (WAF/dedicated network) vs an internet-facing attack surface demanding rigorous authentication and hardening.
- Schema registry enforcing cross-org data contracts vs added governance friction coordinating compatible changes between separate organizations.
- Real-time throughput and observability vs the complexity of distributed tracing and monitoring consumer lag across trust boundaries.
Real-World Example
Consider a regional electricity distribution operator that streams smart-meter readings to retail suppliers, aggregators, and settlement bodies through a cloud-native ESP. Head-end systems collecting interval consumption publish immutable meter-reading events (Protobuf, validated against a shared schema registry) to per-grid-region topics via the Kafka protocol over TLS. Producer traffic crosses the public internet through a WAF in the perimeter subnet, while high-volume settlement agencies connect over dedicated private links. Each supplier's consumer subscribes to relevant topics independently, tracking its own offsets, so a billing platform outage resumes from durable storage without losing readings. When MQTT-connected meters spike during peak demand, the platform absorbs the load. Consumer-lag metrics and distributed tracing let the operator spot a stalled aggregator and confirm delivery across the boundary, all without any direct coupling between metering and external retail systems.
Additional Details
- Event Protocol & Format: Utilize industry-standard streaming protocols (e.g., Kafka protocol, AMQP, MQTT) over TLS. Event data should adhere to common data formats like JSON, Avro, or Protobuf, potentially using schema registries for validation.
- Schema Management: Implement a schema registry to enforce data contract evolution and ensure compatibility between producers and consumers.
- Event-Driven Design: Design event structures and topics following principles of loose coupling and high cohesion, ensuring events are immutable and represent state changes.
- Observability: Enable comprehensive logging, metrics (message rates, latency, errors, consumer lag), and distributed tracing for all ESP components to monitor data flow, platform health, and troubleshoot issues across boundaries.
- Scaling & Resilience: The ESP should be designed for high availability and scalability, allowing independent scaling of producers, topics/partitions, and consumers to handle varying loads and ensure continuous data flow.
- Hybrid Integration: For scenarios involving on-premises systems, integrate the cloud-native ESP with existing enterprise service buses or message brokers using dedicated network connections and secure gateways.
Security Controls
- Perimeter Security: External ingress to the Event Streaming Platform must be secured via a Web Application Firewall (WAF) within a Public Subnet (Perimeter) or a dedicated Virtual Private Cloud (VPC) for ingress. Implement IP whitelisting for known external producers/consumers.
- Transport Security: Enforce strict Transport Layer Security (TLS 1.2 or higher) for all data in transit between external systems and the platform, and internally within the platform components.
- Authentication & Authorization:
- Authenticate external producers and consumers using robust mechanisms such as OAuth 2.0 (Client Credentials Grant), Mutual TLS (mTLS), or API Keys with strong secret management practices.
- Implement granular authorization policies to control which producers can publish to specific topics and which consumers can subscribe to particular event streams.
- Network Connectivity: For highly secure or high-throughput scenarios, leverage dedicated network connections (e.g., Direct Connect, ExpressRoute, Dedicated Interconnect) to establish private, secure links between external networks and the cloud environment hosting the platform.
- Data Encryption: Ensure data at rest within the Event Streaming Platform is encrypted using platform-managed or customer-managed encryption keys.