Pattern Library
A curated library of reusable architecture patterns for enterprise integration and data platforms. Use the search bar, category filters, and tags below to find and apply proven design solutions.
33 integration patterns: 11 integration styles, each for cloud/SaaS, external partner and internal boundaries.
Archived Storage
Retaining petabytes of rarely accessed historical and compliance data on primary high-performance storage drives exorbitant infrastructure costs. Without tiering, per-GB hot-storage fees compound as archives grow, inflating the platform bill while capacity that could serve active workloads is consumed by dormant data.
Data Mart
Exposing an entire enterprise data warehouse directly to business users buries them in irrelevant tables and complex global schemas. Without curation, analysts waste time deciphering structures they don't need, query performance degrades under mixed workloads, and trust in the platform erodes.
Direct Query Mode
Copying entire datasets into BI tool memory yields stale snapshots, hits memory limits, and scatters sensitive data outside the governed warehouse. Refreshing these extracts strains pipelines, dashboards diverge from reality, and audit gaps widen as copies proliferate beyond central controls.
Document Store
Rigid relational schemas struggle to store and query rapidly evolving, hierarchical, or semi-structured data whose shape varies per record. Forcing such data into normalised tables triggers costly migrations, brittle joins, and object-relational mapping overhead that slows iteration.
Dynamic Data Masking
Maintaining physically separate, redacted copies of datasets for each user clearance level is expensive and drives uncontrolled data sprawl. Duplicate copies multiply storage costs, drift out of sync, and widen the attack surface as sensitive data is replicated across many locations.
Federated Query (Query-in-Place)
Physically consolidating every dataset into one central warehouse before it can be queried is slow, costly, and duplicates data. By the time reports run they reflect stale snapshots, engineering teams drown in fragile pipelines, and storage plus data-movement costs steadily escalate.
Geospatial Store
Standard B-tree indexes cannot efficiently resolve multi-dimensional proximity, boundary-intersection, or distance queries over spatial data. Without spatial indexing, these operations degrade into full-table scans with costly per-row geometry math, inflating latency and compute costs as data volumes grow.
Graph Store
Traversing deep, many-to-many relationships in a relational model forces exponentially costly recursive JOINs that degrade as connections grow. As connections multiply, query latency, index bloat, and engineering complexity spiral, making real-time relationship analysis and pattern detection effectively impossible.
GraphQL Data API
REST APIs force clients to over-fetch unused fields or under-fetch, chaining multiple round-trips to assemble linked, deeply nested data models. Without a unified query layer, teams accumulate brittle endpoint sprawl, inflated bandwidth costs, and latency that degrades the user experience as data relationships grow.
Integration AI / ML Services (Cloud)
An internal workload needs to enrich transactions using a managed third-party AI service (a hosted large language model, vision, or speech endpoint) that executes outside the organisation's trust boundary and whose model version, availability, and pricing are controlled by the provider. Sending raw business records to a provider-controlled endpoint risks disclosing regulated content, while unbounded token consumption turns each inference call into an uncapped cost, latency, and availability dependency.
Integration AI / ML Services (External)
An organisation must exchange training datasets and real-time predictions with external third-party partners across trust boundaries. Without controlled ingestion, validation, and serving paths, internal machine learning pipelines and private feature stores risk direct exposure to the public internet, inviting data poisoning, leakage, and unauthorised model access.
Integration AI / ML Services (Internal)
An organisation runs machine learning pipelines that ingest internal training data and serve predictions to internal workloads. Without strict isolation, exposing these datasets, training jobs, or inference endpoints to the public internet risks data leakage, unauthorised access, and compliance violations.
Integration API Management (Cloud)
An internal application must consume or expose synchronous APIs across a public cloud or SaaS boundary, where the counterparty is a managed service outside the organisation's network perimeter and its endpoints, credentials, and rate limits are controlled entirely by the provider. Without a governed egress path and a resilience layer, every SaaS dependency becomes an unmonitored outbound connection and a single point of failure for the calling workload.
Integration API Management (External)
An internal application must synchronously exchange data or services with an external user, partner, or system across a trust boundary in real time. Without a governed edge, inbound and outbound traffic bypasses perimeter controls, exposing internal systems to untrusted networks and leaving connections unsecured, unfiltered, and unlogged.
Integration API Management (Internal)
Internal applications and microservices must exchange data and invoke each other in real time without exposing endpoints to the public internet. Without governed private connectivity, teams resort to ad hoc calls, inconsistent authentication, and unmonitored traffic that erode security, reliability, and observability across the estate.
Integration Change Data Capture (Cloud)
A cloud-hosted system must send row changes, in near real time, to consumers it does not run: partner platforms, other clouds or on-premises applications. They connect over the public internet, so none of them can have a database connection, and traffic to an outside target leaves through a managed egress gateway that only allows approved FQDNs. The source application usually cannot be changed to emit events.
Integration Change Data Capture (External)
A business partner needs a feed of changes from one of your internal data stores, and the partner owns the receiving end: its API, message queue or SFTP server. Deliveries leave your private subnet through a perimeter egress gateway and authenticate to the partner with mTLS, OAuth 2.0 client credentials or SSH keys; the partner never gets a path into your database. Full-table exports or frequent polling load the source, risk its operational integrity and still leave the partner with stale data.
Integration Change Data Capture (Internal)
One internal team needs the changes from another team's database, to keep its own store in sync, drive event processing or feed analytics. Both ends run in the same organisation's private subnets, so traffic stays on the private network and access rests on workload identities (managed identities or service accounts), IAM policies and network segmentation. The changes have to flow quickly while the source system's performance, schema and application code stay untouched.
Integration ETL / Batch Processing (Cloud)
Data teams building a cloud analytics platform must pull data from systems they do not run: on-premises databases, SaaS applications and third-party APIs. It arrives through a perimeter ingestion gateway or a private VPN/Direct Connect-equivalent link, and the pipelines reach cloud data services over private endpoints with managed service identities. Loading that does not scale stalls throughput, raises costs, and lets lapses in data integrity and governance reach downstream analytics.
Integration ETL / Batch Processing (External)
A backend workload and a partner's system must swap large datasets on a schedule, and the partner owns its end of the exchange. Files and API calls cross the internet into a perimeter subnet, where a WAF inspects inbound traffic, the partner authenticates with mTLS, OAuth 2.0 client credentials or SSH keys, and outbound data leaves through an egress gateway. Real-time sync is not needed, but bulk loads that skip validation corrupt records and erode trust between the systems.
Integration ETL / Batch Processing (Internal)
One internal system needs bulk data from another on a regular schedule, and that data changes rarely. Both ends sit in the same organisation's Private Subnet (Workloads), so the transfer never touches the public internet, and the integration service reaches each store with a least-privilege managed identity or service account. Ad hoc or row-by-row movement overloads both systems, breaks consistency and fails to scale as volumes grow.
Integration Event Streaming (Cloud)
Two external systems or organisations must exchange data in real time, but direct peer-to-peer integration is impractical or undesirable. Without an intermediated streaming layer, brittle point-to-point links create tight coupling and unreliable delivery, leaving events lost, delayed, or duplicated across organisational boundaries.
Integration Event Streaming (External)
Partners and third-party systems that your organisation does not run need to publish events to your backend workloads and subscribe to your outbound feeds, in real time. Their traffic crosses the public internet into your perimeter, so each producer and consumer has to pass a WAF and authenticate (OAuth 2.0 client credentials or mutual TLS) before it can reach an internal topic. When no governed path exists, partner event flows bypass these controls or turn into brittle point-to-point links that cannot authenticate each party or scale.
Integration Event Streaming (Internal)
Multiple internal backend workloads in a private subnet must exchange data and event notifications asynchronously without tight coupling. Without a shared messaging backbone, services resort to direct synchronous calls that create brittle dependencies, block on slow peers, and cannot scale or absorb traffic bursts.
Integration Manual / Ad-hoc Transfer (Cloud)
Two disparate systems, often external to the primary cloud, must exchange data at exceptionally low volume and frequency. Building a fully automated integration pathway for such sporadic transfers is economically and operationally prohibitive, leaving the exchange without a defined, secure, or auditable process.
Integration Manual / Ad-hoc Transfer (External)
An internal backend workload must exchange data with an external partner or system only occasionally, at low or sporadic volume. At this volume a fully automated integration is not economically viable, yet ad hoc transfers risk data exposure, lost accountability, and compliance gaps.
Integration Manual / Ad-hoc Transfer (Internal)
Some internal data exchanges between backend workloads occur too rarely to justify a fully automated integration. Without a defined approach, teams either build costly programmatic pipelines that sit idle or fall back on ad hoc, unrepeatable transfers that risk errors and data loss.
Integration Master Data Management (Cloud)
Partners, data providers and downstream applications outside the organisation all use the same customer, product and supplier records, and each keeps its own copy. They reach the cloud-hosted master data over the Public Internet, so traffic must enter through a protected edge and leave through a managed egress gateway limited to allow-listed FQDNs. When the copies drift, business processes break on mismatched values and one bad record spreads through every integration.
Integration Master Data Management (External)
Partner applications and SaaS platforms run by other organisations need to read and update the same customer, product and location records as internal workloads, in near real time. Their calls arrive over the Public Internet, so each partner must authenticate (OAuth 2.0 or mutual TLS) at a WAF-protected API Gateway in the perimeter before reaching the MDM platform. When that path is missing or unreliable, the copies drift and downstream decisions rest on stale or conflicting values.
Integration Master Data Management (Internal)
Teams running internal backend workloads and microservices each create and update the same customers, products and suppliers in their own stores, so the records disagree. Every system belongs to the organisation and traffic stays on its private network, but access to master data is still limited to authorised private subnets and to workloads with a verified identity (mTLS, OAuth 2.0 client credentials or IdP-managed service accounts). The divergent records slow operations, feed decisions with conflicting values and make compliance hard to demonstrate.
Integration Middleware Services (Cloud)
An organisation exchanging real-time data with several third-party systems, each run by another company with its own protocol, ends up with a mesh of point-to-point links and credentials. That traffic crosses the Public Internet (or a dedicated private link) to the cloud, so callers must authenticate at a WAF-protected edge and outbound calls must leave through a managed egress gateway with a static IP and an FQDN allow-list. Direct links are often not technically possible, couple the systems tightly, and leave nowhere for orchestration and transformation.
Integration Middleware Services (External)
A team whose backend workload exchanges data with a partner finds that a direct API is not enough: the partner uses another protocol or format, or needs asynchronous messaging, in both directions. The partner owns the far end and connects over the Public Internet, so its traffic must stop at a WAF and API Gateway (or message proxy) in the perimeter, authenticate with OAuth 2.0 or mutual TLS, and never touch the internal Message Broker. Built into each workload, that mediation gets rebuilt for every partner feed, with no shared governance.
Integration Middleware Services (Internal)
Teams running internal backend workloads need to call each other's services and share events, but their systems use different protocols and formats (REST, SOAP, gRPC, AMQP). Every system belongs to the organisation and traffic stays on its private network, so the controls are workload identity (mTLS or OAuth 2.0 client credentials) and segmentation that limits which workloads reach the integration layer. Wired pair by pair, the links multiply, each carries its own translation and orchestration code, and near-real-time delivery becomes fragile.
Integration Native Connectors (Cloud)
Two cloud applications, or a cloud application and a SaaS platform, need to exchange data, and both already ship an integration mechanism for it. If nobody agrees to use those built-in mechanisms, each team builds its own middleware for the exchange, and the organisation pays for it in cost, latency and upkeep.
Integration Native Connectors (External)
An internal workload in a private subnet must call a system that an external party owns, synchronously over the public internet, using that system's own SDK or connector. Wired up ad hoc, those calls open insecure connections that bypass network isolation and egress controls.
Integration Native Connectors (Internal)
Two backend workloads your organisation runs in the same private subnet need to call each other, and both already have SDKs or client libraries for it. The calls never leave the private network, where TLS, service credentials (OAuth 2.0 or JWTs) and security groups protect them, so forcing them through a dedicated integration platform only adds latency, cost and operational overhead.
Integration Secure File Transfer (Cloud)
Partners, customers and third-party systems you do not control need to exchange large files or batch data with your cloud environment, over the public internet or a dedicated interconnect, so every connection must enter through a WAF-protected perimeter and authenticate with SSH keys or OAuth 2.0. Their capabilities and security posture differ from yours and the data is batch by nature, so direct integration is often impractical, yet every transfer must be secure, reliable and auditable.
Integration Secure File Transfer (External)
External users, partners and systems your organisation does not control need to send and collect files or bulk data over the public internet, so they must stop at a WAF-protected gateway in your perimeter and authenticate with SSH keys, API keys or an identity provider. If they can reach internal services directly, files can be intercepted, unauthenticated parties can get in, and a large transfer that fails partway cannot be reliably recovered.
Integration Secure File Transfer (Internal)
Backend workloads that your teams run in private subnets hand files to each other for batch processing, data sync and event-driven work, so the controls are workload identity (IAM roles, service accounts or SSH keys) and encryption in transit and at rest. When each team moves files its own way, integrity checks get skipped, the audit trail breaks, and files can be intercepted or lost even though they never leave the private network.
Integration Workflow & Orchestration (Cloud)
A business process runs across several of your cloud microservices and also calls partner APIs or data sources that other organisations own, over connections that leave your cloud network through a managed egress gateway that only allows approved FQDNs. When a step fails partway, teams struggle to keep the process transactional, recover it and see where each instance stands.
Integration Workflow & Orchestration (External)
A business process is started by, or has to call, systems owned by partners and third parties, with traffic crossing the public internet in both directions, so inbound calls must stop at a WAF-protected edge API gateway and each partner must authenticate (OAuth 2.0 or mTLS). It also runs across several internal microservices for minutes or days, so teams must keep its state, recover from failures in systems they don't control, and trace each instance end to end.
Integration Workflow & Orchestration (Internal)
Your organisation runs long business processes across several of its own backend workloads and microservices, reached through an internal API Gateway or message broker, with occasional calls to external systems, and each service call authenticates with OAuth 2.0, mTLS or short-lived service credentials. When a step fails, the process loses progress, services are left with inconsistent data, and nobody can reliably see where it stopped or recover it.
Master Data Management (MDM)
Disparate systems each maintain their own version of core business entities like Customer and Product, so no single authoritative record exists. The resulting duplicates and conflicting attributes inflate costs, corrupt analytics and reporting, and expose the business to compliance and regulatory risk.
Materialized Views
Recomputing complex multi-table joins and aggregations on every dashboard request drives up query latency and compute costs. As user concurrency and data volume grow, repeated on-the-fly scans throttle interactive analytics and inflate query-driven billing unpredictably.
Medallion Architecture
Analysts and data engineers working on a lake where every source lands in a single layer, with no quality tiers, end up with a 'data swamp' of unreliable, hard-to-query data. Analysts stop trusting the numbers, pipelines break when a source schema drifts, and when an error surfaces downstream there is no clean raw history to replay or audit.
Open Table Format
Object storage alone lacks ACID transactions, schema evolution, and time-travel, making concurrent writes to large tables error-prone. Without transactional metadata, teams face partial reads, lost updates, expensive full rewrites for schema changes, and no reliable way to audit or roll back historical states.
Retrieval-Augmented Generation (RAG)
Large Language Models lack access to private enterprise data and confidently fabricate plausible but false answers about internal topics. Without grounding in authoritative sources, users receive unverifiable responses that erode trust, drive costly errors, and expose the organisation to compliance risk.
Star Schema
Highly normalised relational schemas force analytical queries through many complex joins, crippling performance on large-scale reporting. As data volumes grow, dashboards slow to a crawl, query costs escalate, and business users struggle to explore data without deep SQL expertise.
Time-Series Store
General-purpose databases buckle when ingesting millions of high-frequency timestamped events and computing temporal aggregations. Without a purpose-built engine, ingestion back-pressures producers, storage costs balloon, and time-window queries slow to a crawl, undermining real-time dashboards and alerting.
Unified Batch & Stream Processing (Kappa)
Maintaining separate batch and stream processing systems forces teams to implement the same business logic twice in divergent codebases. Over time these paths drift, producing conflicting results between historical and real-time views while doubling infrastructure, testing, and operational costs.
Webhooks (Event-Push Ingestion)
Polling source APIs on a fixed schedule adds latency, wastes bandwidth on empty responses, and cannot keep pace with real-time event streams. Without a push model, ingestion pipelines fall behind, incur rising API costs, and hit rate limits that force teams to choose between freshness and reliability.