Integration Event Streaming (External)

Partners and third-party systems that your organisation does not run need to publish events to your backend workloads and subscribe to your outbound feeds, in real time. Their traffic crosses the public internet into your perimeter, so each producer and consumer has to pass a WAF and authenticate (OAuth 2.0 client credentials or mutual TLS) before it can reach an internal topic. When no governed path exists, partner event flows bypass these controls or turn into brittle point-to-point links that cannot authenticate each party or scale.

Run an Event Streaming Platform inside your network, with dedicated topics or streams for each asynchronous exchange. External producers connect to an ingress event proxy in the Public Subnet (Perimeter), which forwards their events to internal topics. External consumers subscribe to internal topics or queues through an egress event proxy or a managed egress gateway in the same perimeter, and only authenticated parties can publish or subscribe.

Data Ingestion Flow (External Producer to Internal ESP):

1. External Producer: A third-party system or partner publishes streaming data (e.g., messages, events) over a secure channel (e.g., TLS-enabled TCP or HTTPS) to the designated ingress endpoint.
2. Web Application Firewall (WAF) / Event Proxy Ingress: The traffic first hits a public-facing Web Application Firewall (WAF) for initial security screening. Subsequently, a secure Event Proxy (Ingress) in the Public Subnet (Perimeter) receives the authenticated data stream, terminates TLS, and forwards it to the internal Event Streaming Platform.
3. Event Streaming Platform (ESP): The internal ESP ingests the data into a configured topic or stream. Access control (ACLs, RBAC) is enforced at this layer.
4. Backend Workload: Internal microservices or data processing pipelines subscribe to the ESP topic to consume, process, and act upon the ingested streaming data in real-time.

Data Publication Flow (Internal ESP to External Consumer):

1. Backend Workload: An internal microservice publishes data or events to a topic on the Event Streaming Platform.
2. Event Streaming Platform (ESP): The ESP manages the published data and makes it available to authorised subscribers.
3. Event Proxy Egress / Managed NAT: For external consumption, the ESP or a dedicated internal service routes the data stream to an Event Proxy (Egress) or a Managed NAT/Egress Gateway in the Public Subnet (Perimeter). This egress point applies security policies (e.g., authentication, FQDN whitelisting) and performs any necessary network address translation.
4. External Consumer: The authorised external system subscribes to the designated secure egress endpoint (e.g., through a stream API, managed queue, or dedicated endpoint) to receive the streaming data.

• Event-Driven Architecture (EDA)
• Asynchronous Messaging & Stream Processing
• Edge Security (Ingress/Egress control for event streams)
• Managed Service Consumption (for Event Streaming Platforms)
• Zero Trust Principles

• Perimeter Security: All external connections (inbound or outbound) must traverse a dedicated secure proxy or gateway deployed within the Public Subnet (Perimeter). Protect this gateway with a Web Application Firewall (WAF) to filter malicious traffic and prevent unauthorised access attempts.
• Transport Security: Mandate strict Transport Layer Security (TLS 1.2 or higher) for all data streams, ensuring end-to-end encryption for both inbound data ingestion and outbound data publication.
• Authentication & Authorisation:
  • For system-to-system (B2B) integrations, authenticate external event producers and consumers using mechanisms such as OAuth 2.0 (Client Credentials Grant) or Mutual TLS (mTLS).
  • Use access control lists (ACLs) or role-based access control (RBAC) on the Event Streaming Platform to enforce fine-grained authorisation for topic access (publish/subscribe).
• Traffic Management: Implement flow control mechanisms on the event proxy or Event Streaming Platform, such as rate limiting, throttling, and connection quotas, to prevent resource exhaustion from high-volume external traffic.
• Outbound Controls: When internal workloads publish data to external subscribers or systems, route all outbound streaming traffic through a managed Egress Gateway or dedicated event proxy. Configure strict egress filtering (IP whitelisting, FQDN whitelisting) and, if required, perform Data Loss Prevention (DLP) inspection on outbound streams.
• Data Integrity: Implement mechanisms to ensure data integrity, such as message signing or checksums, where applicable for critical data streams.

• Real-time, decoupled cross-boundary streaming vs the operational burden of running an ESP plus ingress and egress proxies.
• Strong perimeter security via WAF, TLS termination, and authenticated proxies vs added latency and hops on every message.
• Durable, replayable event streams for external integrators vs the storage cost and stream governance discipline required to prevent breaking changes.
• Independent scaling of external producers and consumers vs the complexity of managing credentials, ACLs, and schema versioning across many parties.
• Standardised protocols and contracts (AsyncAPI, CloudEvents) vs the upfront effort to document, version, and maintain those interface specifications.

• Protocols & Formats: Use open streaming protocols (e.g., Kafka protocol, AMQP, MQTT over TLS, WebSocket). Data formats should be standardised (e.g., JSON, Avro, Protobuf) and adhere to common data models or industry standards. Event metadata should conform to specifications like CloudEvents.
• Documentation & Specification: Publish detailed interface documentation for external producers and consumers, using specifications such as AsyncAPI for event-driven architectures. Define clear schema for data objects exchanged.
• Credential Management: Implement a mechanism for securely managing credentials (e.g., API keys, OAuth client secrets, mTLS certificates) for all external parties interacting with the Event Streaming Platform.
• Observability: Enable comprehensive logging, metrics (message throughput, latency, error rates), and distributed tracing (e.g., OpenTelemetry) across the entire streaming pipeline, including the event proxies and the Event Streaming Platform, to monitor cross-boundary data flows and quickly diagnose issues.
• Stream Governance: Implement strict topic/stream naming conventions, versioning strategies, and lifecycle management for event streams to maintain consistency and avoid breaking changes for external integrators. Treat ESP configurations as code (IaC/GitOps) for repeatable deployments.

A global parcel carrier streams shipment tracking events across its partner network using this pattern. Retail shippers and customs brokers publish scan and manifest events over TLS to a public ingress endpoint, where a WAF screens traffic before a secure Event Proxy terminates TLS and forwards messages onto internal Kafka-protocol topics with RBAC enforced. Each shipper authenticates with OAuth 2.0 client credentials, and the proxy applies a rate limit per partner, so one shipper's end-of-day burst cannot starve the others. Route-optimisation and delivery-estimation microservices consume these streams in real time. Backend workloads publish enriched status updates (out-for-delivery, exception and proof-of-delivery events) back to an ESP topic. An egress Event Proxy then applies mTLS authentication and FQDN allow-listing, and each e-commerce marketplace subscribes to the same status stream through it and reads at its own pace. When a marketplace's consumer is offline overnight, it replays the retained events in the morning and catches up without losing a scan. AsyncAPI specifications and Avro schemas govern every payload, while OpenTelemetry tracing measures throughput and latency across the perimeter so operators can catch backlogs before customer-facing tracking pages lag.