Integration Master Data Management (Cloud)
Problem
Partners, data providers and downstream applications outside the organisation all use the same customer, product and supplier records, and each keeps its own copy. They reach the cloud-hosted master data over the Public Internet, so traffic must enter through a protected edge and leave through a managed egress gateway limited to allow-listed FQDNs. When the copies drift, business processes break on mismatched values and one bad record spreads through every integration.
Solution
Run a cloud-native Master Data Management (MDM) platform as the single source of truth for critical reference data. Matching, merging and survivorship rules maintain the golden records. External systems use APIs for synchronous requests and event streams for asynchronous updates, always through an API Gateway or Message Broker at the edge, so every exchange is authenticated, controlled and auditable.
Cloud Paradigm
- Data Fabric / Data Mesh principles for distributed data governance
- API-First Design for controlled data access
- Event-Driven Architecture for real-time data synchronisation
- Data Streaming for continuous data distribution
- Zero Trust Architecture for secure cross-boundary interactions
- Hybrid Integration Platform for connectivity across cloud and on-premises environments
Solution Flow
Data Ingestion Flow (External to MDM):
- External Source System: An external system (e.g., partner application, data provider) sends new or updated reference data.
- Web Application Firewall (WAF) / API Gateway / Message Broker: Traffic crosses the Public Internet to a WAF-protected ingress point. An API Gateway authenticates, authorises and routes synchronous updates. A cloud-native Message Broker (e.g., a managed queue or streaming service) receives asynchronous events and messages for reliable delivery.
- Data Ingestion Workload: An ingestion service or microservice in the Private Subnet (Workloads) consumes the data from the API Gateway or Message Broker, validates it, runs data quality checks and may apply light transformation.
- MDM Platform: The Master Data Management (MDM) platform applies its mastering rules (e.g., matching, merging, survivorship) and creates or updates the golden record.
- Audit & Response: The MDM platform records the transaction for audit, then sends an acknowledgement (synchronous requests) or publishes a "golden record updated" event (asynchronous flows).
Data Distribution Flow (MDM to External):
- MDM Platform (Golden Record Update): A new or changed golden record triggers a distribution event, or the MDM platform makes the data available.
- Data Distribution Workload / Message Broker:
- Asynchronous Push: The MDM platform or a distribution service publishes the update event to a cloud-native Message Broker (e.g., pub/sub topic), and external consuming systems subscribe to the topics.
- Synchronous Pull: An API endpoint exposed through an API Gateway gives read access to the golden records.
- Egress Gateway / API Gateway: The Message Broker delivers pushed events; for pulls, the external system queries the API Gateway. Any direct outbound connection from the MDM environment to an external system goes through a Managed NAT / Egress Gateway for filtering and control.
- External Consuming System: The external system receives the update by API call or as a message from the Message Broker, then updates its local data stores.
When to Use
- Several external systems (partners, data providers, downstream apps) must use the same authoritative reference data (customers, products, suppliers) without their copies drifting.
- You need synchronous reads and asynchronous change propagation from one mastered source.
- Regulation or contracts require auditable, mediated data exchange, not direct database access.
- Golden records depend on matching, merging and survivorship across conflicting inbound sources.
- The systems span organisational boundaries, where edge control (WAF, API Gateway, Message Broker) is mandatory.
When NOT to Use
- One application owns and consumes its data and shares it with nothing else: an application database is enough.
- Reference data is small, static and rarely changes; a versioned config file or shared library is simpler.
- You only need point-to-point data movement, with no mastering, matching or golden-record logic: use a plain integration or ETL pattern.
- You need ultra-low-latency, high-frequency transactional data; MDM mastering adds unacceptable delay.
- Sources cannot agree on governance, stewardship or survivorship rules, so authoritative mastering is not politically or organisationally viable.
Trade-offs
- Single authoritative source of truth vs the governance overhead of stewardship, data-quality rules and survivorship configuration.
- Decoupled, resilient exchange via API Gateway and Message Broker vs more infrastructure and edge components to run and secure.
- Dual synchronous/asynchronous interfaces vs maintaining versioned API contracts and event schemas in parallel.
- Strict mediated access with no direct DB connections vs slower onboarding for external systems, which must use managed interfaces.
- End-to-end observability and auditability vs the cost of centralised logging, tracing and metrics across every integration hop.
Illustrative Scenario
A hypothetical scenario showing how the pattern applies. Not a client case study.
Picture a hotel group, invented for this example, that runs several brands and keeps its guest, property and rate-plan reference data in a cloud-native MDM platform. Channel partners and online travel agencies (OTAs) send updated property and room-type details as events across the Public Internet. The events land in a WAF-protected Message Broker. An ingestion microservice in the private subnet validates them and runs data-quality checks, then submits them to the MDM, which applies matching, merging and survivorship rules to master each golden record.
On every change, the platform publishes a 'golden record updated' event to a pub/sub topic. The central reservation, loyalty and property-management systems subscribe to it and update their local stores. A partner extranet pulls current rate plans synchronously through the API Gateway. Every interaction is authenticated, mediated and logged, so each change to a golden record can be traced to the partner and event that made it.
Additional Details
- MDM Style & Strategy: The MDM style (e.g., Registry, Consolidation, Coexistence, Centralised) sets how complex the integration is and which interfaces it needs.
- Data Governance & Quality: Run data stewardship, data quality rules and validation inside the MDM platform to keep reference data correct.
- API Design & Event Schema: Publish versioned API contracts (e.g., OpenAPI Specification) for synchronous access and standard event schemas (e.g., AsyncAPI, Avro) for asynchronous distribution, in consistent formats (e.g., JSON, XML).
- Error Handling & Idempotency: Use error handling, retries with exponential backoff and idempotency in both ingestion and distribution, so failed or repeated calls leave data consistent. Configure survivorship rules in the MDM for conflicting updates.
- Observability: Collect centralised logs, metrics (e.g., data volume, latency, error rates, mastering success rates) and distributed traces across the API Gateway, Message Broker, ingestion and distribution workloads, and MDM platform.
- Security by Design: External parties never connect directly to the MDM database; every interaction goes through secure, managed interfaces.
- Infrastructure as Code (IaC): Deploy all MDM components, integration interfaces and security configuration through IaC and CI/CD pipelines, so every environment is consistent and repeatable.
Security Controls
- Perimeter Security: Inbound connections from external systems terminate at an Edge API Gateway or a cloud-native Message Broker in a Public Subnet (Perimeter), behind a Web Application Firewall (WAF) that blocks common attacks (e.g., OWASP Top 10) and manages traffic.
- Transport Security: Enforce TLS 1.2 or higher on all data in transit. Consider Mutual TLS (mTLS) for system-to-system exchanges of highly sensitive data.
- Authentication & Authorisation:
- Authenticate external systems in system-to-system (B2B) integrations with OAuth 2.0 (Client Credentials Grant), Mutual TLS (mTLS) or signed messages.
- Apply fine-grained authorisation (e.g., attribute-based access control, RBAC) in the MDM platform and at the API Gateway to limit access to specific data sets or operations.
- Traffic Management: Use API Management policies or Message Broker settings for rate limits, throttling and burst quotas, protecting the MDM backend from traffic spikes and Denial of Service (DoS) attacks.
- Data Encryption: Encrypt sensitive reference data at rest in the MDM platform and its data stores.
- Outbound Controls: Route outbound connections from the MDM platform or its integration components through a Managed NAT / Egress Gateway that only permits listed external IP addresses or fully qualified domain names (FQDNs). Add Data Loss Prevention (DLP) scanning on outbound streams if compliance requires it.
- Credential Management: Never store credentials in cleartext; keep API keys, database credentials and other secrets in a secrets management service.
Related Patterns
Use and cite this
Free to use and adapt under CC BY 4.0. Credit: David Tirabassi, IT Architecture Patterns.
Attribution: Source: David Tirabassi, IT Architecture Patterns (https://www.itarchitecturepatterns.net/patterns/int-mdm-cloud), licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).