Integration Master Data Management (Internal)
Problem
Teams running internal backend workloads and microservices each create and update the same customers, products and suppliers in their own stores, so the records disagree. Every system belongs to the organisation and traffic stays on its private network, but access to master data is still limited to authorised private subnets and to workloads with a verified identity (mTLS, OAuth 2.0 client credentials or IdP-managed service accounts). The divergent records slow operations, feed decisions with conflicting values and make compliance hard to demonstrate.
Solution
Make a central Master Data Management (MDM) platform the source of truth for key business entities. Pick integration mechanisms to suit the MDM style (e.g., Registry, Consolidation, Coexistence, Centralised), so data is synchronised, propagated or federated across every internal workload that contributes or consumes it. Define data governance policies, data quality rules and secure exchange patterns with it.
Cloud Paradigm
- Master Data Management (MDM) as a Service
- Data Governance and Data Quality Management
- Event-Driven Architecture (for real-time data propagation)
- Data Consolidation, Federation, and Propagation patterns
- Data-as-a-Service (DaaS) for exposing master data
- Immutable Data principles for auditability
Solution Flow
This pattern follows master data from ingestion and governance to distribution and consumption by internal backend workloads. The exact flow depends on the MDM style (Registry, Consolidation, Coexistence, Centralised).
Data Ingestion & Governance Flow:
- Source System(s) / Data Steward: Master data comes from internal backend workloads, or data stewards enter and curate it.
- Data Ingestion Interface: Data reaches the MDM platform through secure APIs (e.g., REST, GraphQL), batch file transfers (secure file transfer protocols over private networks) or event streams.
- MDM Data Quality & Validation: The Master Data Management platform applies data quality rules, validation checks, standardisation and enrichment. Survivorship rules resolve conflicts between sources.
- Master Data Storage: The validated, cleansed master data is stored in the MDM platform's central repository, the single source of truth.
Master Data Distribution & Consumption Flow:
How validated master data reaches internal consuming backend workloads.
- MDM Platform: With the golden record established, the MDM platform is ready to distribute data.
- Data Distribution Mechanism (Pull/Push):
- Pull-based (Request/Response): Consuming backend workloads make synchronous API calls (e.g., RESTful endpoints exposed by the MDM platform) for specific records on demand.
- Push-based (Event-Driven): The MDM platform publishes master data changes (e.g., create, update, delete events) to a secure message broker or event streaming platform. Consuming workloads subscribe to relevant topics and receive updates in near real time.
- Batch-based: For large datasets or less time-sensitive cases, the MDM platform writes secure batch exports (e.g., CSV, Parquet files) to Object Storage for consuming systems to ingest.
- Consuming Backend Workload(s): Internal microservices or applications query the MDM, process events or ingest batch files, and update their local caches or transactional data stores.
When to Use
- The same business entity (customer, product, supplier) is created and updated separately in several internal systems, and the records diverge.
- Regulatory or reporting obligations demand a demonstrable, auditable golden record with enforced survivorship rules.
- Downstream analytics or operational decisions rely on stale or conflicting attribute values.
- You want data quality, standardisation and enrichment applied once, centrally, not rebuilt in every service.
- Several consumers need the same entity through different access styles: on-demand API lookup, event subscription and bulk batch loads.
When NOT to Use
- One service owns the entity data end to end and nothing else consumes it: a service-owned datastore is simpler.
- The data is high-volume transactional or event data (orders, telemetry), not slowly changing reference entities; use streaming or CQRS patterns.
- Writes need near-zero latency; MDM validation, survivorship and propagation add processing time.
- A lightweight cross-reference is enough: a Registry-style index or shared lookup service may do without full consolidation.
- Nobody will own governance; with no data stewards, the platform becomes one more unauthoritative copy.
Trade-offs
- Single authoritative golden record vs the organisational overhead of governance roles, stewardship workflows and survivorship rule maintenance.
- Consistent quality and enrichment applied centrally vs added ingestion latency from validation and standardisation steps.
- Decoupled consumers via events and APIs vs a new critical dependency whose outage hits many downstream workloads.
- Conflict resolution across sources vs the complexity of tuning survivorship rules and handling dead-letter queues and reconciliation.
- Flexible distribution styles (pull, push, batch) vs eventual-consistency windows in which local caches differ from the master.
Illustrative Scenario
A hypothetical scenario showing how the pattern applies. Not a client case study.
Imagine a mobile network operator, invented for this example, whose subscriber identity lives separately in its CRM, billing engine, provisioning platform and service-activation microservices. Each holds slightly different names, SIM associations and contact numbers, and none of them is authoritative.
The operator adopts a Coexistence-style MDM. Each system submits records through secure REST ingestion. Data quality rules standardise MSISDNs and account identifiers, and survivorship rules collapse conflicts into one golden subscriber record. When an attribute changes, the MDM publishes create and update events to an event streaming platform. Provisioning and billing subscribe and refresh their local stores in near real time. A nightly Parquet export to object storage reloads the churn-analytics warehouse, and on-demand REST lookups serve the care portal. Dead-letter queues hold unresolved conflicts for data stewards to review. OpenTelemetry traces synchronisation latency and data quality scores across every integration point, so stewards can see which system keeps sending conflicting records.
Additional Details
-
MDM Styles:
- Registry: An index of master data that federates queries to source systems without consolidating the data.
- Consolidation: Consolidates data from several sources into a "golden record" but does not distribute it back to the sources.
- Coexistence: Consolidates data and actively synchronises the golden record back to source and consuming systems.
- Centralised: The primary system for creating and updating master data, which it propagates to every other system.
-
Data Governance & Stewardship: Assign clear roles for data ownership, quality enforcement and master data lifecycle management, with automated workflows for data issues and change requests.
-
Integration Patterns: Choose by latency, data volume and complexity:
- API-led Integration: Synchronous, on-demand lookup of individual records.
- Event-Driven Integration: Real-time propagation of changes to interested subscribers.
- Batch Integration: Bulk synchronisation, initial loads, or cases where near-real-time updates are not critical.
-
Error Handling & Data Survivorship: Use retries, dead-letter queues for events and alerts for data quality issues on every ingestion and distribution process. Configure survivorship rules in the MDM to resolve source conflicts automatically.
-
Observability: Collect logs, metrics (data quality scores, synchronisation latency, record counts) and distributed traces (e.g., OpenTelemetry) for the MDM platform and every integration point, to see data flow, quality and performance.
Security Controls
-
Transport Security: Enforce TLS 1.2 or higher on all data exchanges between the MDM platform and integrated backend workloads.
-
Authentication & Authorisation:
- Authenticate systems with strong workload identity: Mutual TLS (mTLS), OAuth 2.0 (Client Credentials Grant) or Service Accounts managed by an Identity Provider (IdP).
- Apply granular role-based access control (RBAC) in the MDM platform to limit data access and management functions by user or service role.
-
Data Security at Rest: Encrypt all master data in the MDM platform's persistence layer with platform-managed or customer-managed keys.
-
Access Control: Give each backend workload least-privilege access to master data. Use network segmentation so only authorised Private Subnets (Workloads) can reach the MDM platform.
-
Audit & Logging: Log every data change, access attempt and administrative action, and send the logs to a central Security Information and Event Management (SIEM) system for real-time monitoring and anomaly detection.
-
Data Privacy & Compliance: Meet the relevant data privacy regulations with data masking, anonymisation or pseudonymisation where required, especially for sensitive attributes.
Related Patterns
Use and cite this
Free to use and adapt under CC BY 4.0. Credit: David Tirabassi, IT Architecture Patterns.
Attribution: Source: David Tirabassi, IT Architecture Patterns (https://www.itarchitecturepatterns.net/patterns/int-mdm-internal), licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).