Integration Workflow & Orchestration (Internal)
Context
Several of your own backend workloads and microservices, all on your private network, each handle one part of a long-running business process such as an approval or a repair cycle. Process data passes between them over internal routes, with an occasional call out to an external system.
Problem
Your organisation runs long business processes across several of its own backend workloads and microservices, reached through an internal API Gateway or message broker, with occasional calls to external systems, and each service call authenticates with OAuth 2.0, mTLS or short-lived service credentials. When a step fails, the process loses progress, services are left with inconsistent data, and nobody can reliably see where it stopped or recover it.
Forces
- A process can run for days across several services, but each service call commits on its own, so a failure partway leaves data inconsistent.
- Operators need one place to see and recover each process, but a central orchestrator can become a chokepoint and a single point of failure.
- Some steps wait for a person or a slow external reply, but the process must not lose its state while it waits.
- Business rules change often, but hardcoding them in each service means a code change for every rule change.
- Most calls stay on internal routes, but a few steps must reach external APIs through a controlled exit point.
Solution
Run the process on a cloud-native, serverless workflow orchestration engine that persists state at every step, with a decision management service for rule-based branching. Service tasks call internal workloads using workload credentials (OAuth 2.0 client credentials, mTLS or short-lived service accounts), and reach external APIs only through a Managed NAT/Egress Gateway.
Solution Flow
Orchestration Flow (Internal Process Execution):
- Initiation: A trigger (e.g., an API Gateway request, an event from a message broker, a scheduled event, or a human action in a user interface) starts a new instance of the business workflow.
- Workflow Orchestrator: The serverless workflow orchestration engine receives the trigger, initialises the workflow state and runs the defined sequence of steps.
- Decision Management Service (Optional): If a step needs a dynamic decision, the orchestrator passes the relevant data to a decision management service, which evaluates predefined business rules and returns a decision.
- Service Task Execution: The orchestrator runs service tasks, typically:
- Invoking an internal Backend Workload/Microservice through an internal API Gateway (synchronous protocols such as REST, gRPC) or sending a message to a message broker for asynchronous processing.
- Calling external systems through a Managed NAT / Egress Gateway (if outbound connectivity is required).
- Updating data in object storage or databases.
- Human Task Integration (Optional): For steps that need a person (e.g., approvals, data entry), the orchestrator can:
- Send notifications (email, push notification).
- Integrate with a task management system or a web form in a user interface.
- Pause the workflow until the human task is complete.
- State Management & Error Handling: The engine maintains process state, handles retries, compensates for failures and applies predefined error handling (e.g., alerts, alternative paths, rollbacks).
- Completion: When all steps succeed, the workflow updates its status and can emit an event or trigger a follow-on process.
Cloud Paradigm
- Serverless Workflow Orchestration
- Event-Driven Architecture
- Microservices Orchestration and Choreography
- Decision as a Service
- Human-in-the-Loop Automation
- Observability by Design
Security Controls
- Access Control: Restrict configuration and execution of the orchestration engine and decision management service with granular Identity and Access Management (IAM) policies.
- Transport Security: Enforce Transport Layer Security (TLS 1.2 or higher) between the workflow engine, decision service, invoked backend workloads and external APIs.
- Authentication & Authorisation (Service-to-Service):
- When invoking backend workloads (microservices), authenticate with OAuth 2.0 (Client Credentials Grant), Mutual TLS (mTLS), or short-lived credentials/service accounts.
- Apply authorisation policies that limit what each workflow can access or modify in downstream systems.
- Data Protection: Encrypt sensitive workflow state at rest and in transit, and mask or tokenise highly sensitive data the workflow or decision service handles.
- Audit Logging: Log all workflow executions, state changes, decision outcomes and access attempts to a central Security Information and Event Management (SIEM) system.
- Input Validation: Validate and sanitise input at the workflow entry point to prevent injection attacks and protect data integrity.
- Least Privilege: Grant the workflow engine's execution roles only the permissions they need for the resources and services they use.
Builds on
Catalogue patterns this one applies. The names link to their catalogues; the notes on how they are used here are ours.
- Process Manager (opens in a new tab) (Enterprise Integration Patterns): The workflow engine plays this role: it keeps each process instance's state and decides the next step after every reply.
- Saga (opens in a new tab) (Azure Architecture Center): A failed step triggers compensation for the steps already completed, which keeps data consistent across services.
- Compensating Transaction (opens in a new tab) (Azure Architecture Center): Each service task that changes data has an undo step, such as releasing held inventory, which the engine runs when a later step fails.
When to Use
- You orchestrate multi-step processes across several microservices or backend workloads and need durable state at each step.
- Business logic needs explicit error handling, retries and compensation (e.g., Saga-style rollbacks) to keep distributed transactions consistent.
- Processes are long-running and must pause and resume, for example for human approvals or delayed external responses.
- Business rules change often and belong in a decision management service, not hardcoded application logic.
- You need auditable, versioned workflow definitions integrated into CI/CD and observable through distributed tracing.
When NOT to Use
- The interaction is a single synchronous request/response without meaningful intermediate state. A direct API call is simpler.
- High-throughput, low-latency event processing cannot absorb per-execution orchestration overhead; prefer stream processing or event-driven choreography.
- Services can react to events independently. Choreography avoids the orchestrator becoming a bottleneck and single point of governance.
- The workflow is purely data transformation or ETL; a data pipeline or batch framework fits better.
- Steps are trivial and rarely change, so running a workflow engine costs more than it returns.
Trade-offs
- Central visibility and state management vs the orchestrator becoming a governance chokepoint and single point of failure that needs HA design.
- Declarative, versioned process definitions vs modelling discipline and a learning curve for teams used to imperative code.
- Built-in retries and compensation vs the effort of designing idempotent service tasks and correct rollback logic.
- Serverless scaling and less infrastructure to manage vs per-transition execution costs and possible lock-in to the engine's dialect.
- Observability across steps vs the complexity of distributed tracing across the orchestrator and every invoked service.
Implied decisions
- Choose the workflow engine and how definitions are written (JSON, YAML or a graphical model), knowing you may be tied to its dialect.
- Decide which rules live in the decision management service and which stay in the workflow definition.
- Design a compensation step for every task that changes data, and make each service task idempotent so retries are safe.
- Choose, per step, between a synchronous call through the internal API Gateway and an asynchronous message through the broker.
- Choose the service-to-service credential: OAuth 2.0 client credentials, mTLS or short-lived service accounts.
- Decide which steps need a human task, and how the workflow notifies people and waits for them.
Additional Details
- Workflow Definition: Define workflows in declarative languages (e.g., JSON, YAML) or graphical modelling tools, under version control and CI/CD.
- Idempotency: Make service tasks idempotent where possible so transient failures can be retried safely.
- Compensation Logic: Add compensation steps that undo completed work when a critical transaction fails, keeping data consistent (e.g., Saga pattern).
- Scalability & Resilience: Use the built-in scaling and fault tolerance of cloud-native workflow engines to distribute tasks and stay highly available.
- Observability: Trace workflow steps and invoked services (e.g., OpenTelemetry), with central logs and metrics for workflow execution, task duration and error rates.
- Integration Patterns: Invoke backend workloads through API calls via an internal API Gateway, asynchronous messaging via a Message Broker, or event streaming, as each step requires.
- Long-Running Processes: The engine should persist state so workflows can pause and resume over extended periods.
Illustrative Scenario
A hypothetical scenario showing how the pattern applies. Not a client case study.
At a hypothetical open-pit mine, equipment mobilisation runs on a serverless workflow orchestration engine. When a haul truck logs a fault code, an event from the telemetry message broker starts a workflow instance, which keeps its state through a repair cycle that can last several days. A decision management service applies severity and warranty rules to route the request. Service tasks call internal microservices through the internal API Gateway to reserve spare parts and dispatch maintenance crews. A managed egress gateway carries the one external call, which pulls OEM diagnostics from a supplier's API. Every other call stays on internal routes behind the API Gateway. When a shift supervisor must approve downtime, the workflow sends a notification and pauses until that human task is complete. If a parts-reservation step fails, compensation logic releases the held inventory so data stays consistent. On completion, the workflow emits an event that returns the truck to the fleet schedule.
Use and cite this
Free to use and adapt under CC BY 4.0. Credit: David Tirabassi, IT Architecture Patterns.
Attribution: Source: David Tirabassi, IT Architecture Patterns (https://www.itarchitecturepatterns.net/patterns/int-workflow-internal), licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/).