Integration Secure File Transfer (External)
Problem
External users, partners and systems your organisation does not control need to send and collect files or bulk data over the public internet, so they must stop at a WAF-protected gateway in your perimeter and authenticate with SSH keys, API keys or an identity provider. If they can reach internal services directly, files can be intercepted, unauthenticated parties can get in, and a large transfer that fails partway cannot be reliably recovered.
Solution
Put a Managed Secure File Transfer Gateway in the Public Subnet (Perimeter), behind Edge Protection (WAF), as the only thing outsiders connect to. It authenticates each party (SSH key, API key or OAuth/OIDC), stages files in encrypted Object Storage in a private subnet, and sends outbound files through a Managed NAT/Egress Gateway, isolating internal workloads.
Cloud Paradigm
- Asynchronous Data Exchange
- Event-Driven Architecture (for notifications)
- Edge Security (Ingress/Egress control)
- Zero Trust Architecture
- Managed Service Consumption
- Object Storage as a Data Lake Ingestion Point
Solution Flow
Inbound Data Transfer Flow (External to Internal):
- External Party: An external system or user starts a secure transfer (e.g., SFTP, FTPS, HTTPS) over the Public Internet to the Managed Secure File Transfer Gateway.
- Edge Protection (WAF): At the public-facing cloud edge, Edge Protection (WAF) filters malicious requests and common attack patterns.
- Managed Secure File Transfer Gateway: The gateway authenticates the party (e.g., SSH key, API key, OAuth/OIDC), terminates the encrypted session (SSH for SFTP, TLS for FTPS or HTTPS), enforces access policies and stages the file in a designated, encrypted Object Storage bucket in a Private Subnet (Workloads).
- Event Notification (Optional): A successful upload can trigger an event (e.g., message queue, serverless function) that notifies downstream workloads or microservices.
- Backend Workload: An internal workload or microservice (e.g., data processing pipeline, ETL component) retrieves the file to process, validate and integrate it.
Outbound Data Transfer Flow (Internal to External):
- Backend Workload: An internal workload or microservice places the outbound file in a designated, encrypted Object Storage bucket in a Private Subnet (Workloads).
- Managed Secure File Transfer Gateway: The gateway (or a dedicated egress module) retrieves the file on a predefined schedule or event trigger.
- Managed NAT / Egress Gateway: If the gateway connects out from a private subnet, traffic passes through a Managed NAT/Egress Gateway that enforces outbound FQDN allow-listing, performs network address translation and provides a static, allow-listed egress IP.
- External Target: The gateway transfers the file (e.g., SFTP, FTPS, HTTPS) to the partner's system over the Public Internet from a controlled egress point.
When to Use
- You must exchange files with external partners over standard protocols (SFTP, FTPS, HTTPS) and cannot make them adopt your internal APIs.
- The exchange is batch or bulk (nightly reconciliation files, large exports, document dumps), not fine-grained record calls.
- Regulation or contracts demand strong perimeter authentication, static allow-listed egress IPs and full audit trails of every transfer.
- Internal workloads must be fully isolated from external parties, with Object Storage as the only shared handoff point.
- Transfers can be asynchronous, letting producers and consumers run on independent schedules or event triggers.
When NOT to Use
- Interactions are low-latency, request/response or transactional. Use an API Gateway or synchronous integration pattern.
- The exchange is high-frequency event streaming of small messages, better suited to a message broker or streaming platform.
- Both endpoints are internal and under your control. Direct service calls or an internal event bus avoid the gateway overhead.
- Complex transformation or protocol orchestration is the main need. Use a dedicated integration or ETL platform, not the transfer gateway.
- The partner supports only legacy plain FTP or SCP and cannot upgrade; this pattern deliberately excludes those insecure protocols.
Trade-offs
- Strong perimeter isolation and controlled egress vs the cost of running a hardened gateway, WAF and NAT/egress infrastructure.
- Reliable asynchronous decoupling via Object Storage staging vs added end-to-end latency and eventual-consistency handling.
- Standard secure protocols with broad partner compatibility vs the credential lifecycle burden (SSH keys, certificates, rotation) across many parties.
- Full auditability and lifecycle governance vs the storage, retention and monitoring overhead of compliance.
- Clear separation of transfer from transformation vs extra middleware when workflows need enrichment or protocol conversion.
Illustrative Scenario
A hypothetical scenario showing how the pattern applies. Not a client case study.
An open-pit mining operator, in this hypothetical, collects nightly production and equipment-telemetry files from dozens of drilling, haulage and survey contractors. Each contractor connects over SFTP across the Public Internet, and none of them has to adopt the operator's internal APIs. Edge Protection (WAF) filters hostile traffic, then the Managed Secure File Transfer Gateway authenticates the contractor by SSH key, terminates the SSH session and stages the raw file in an encrypted Object Storage bucket in a Private Subnet. The upload fires an Event Notification, which triggers an internal ETL workload to validate tonnage, blast logs and fleet-usage data. After reconciliation, backend pipelines drop settlement and compliance reports into a designated bucket. The gateway collects them on schedule and pushes them to each contractor through a Managed NAT/Egress Gateway with a static, allow-listed egress IP. No contractor ever connects to the core ore-accounting workloads, which stay fully isolated, and Object Storage remains the only shared handoff point.
Additional Details
- Supported Protocols: Use SFTP (SSH File Transfer Protocol), FTPS (FTP over TLS/SSL) or HTTPS-based uploads and downloads. Never use legacy protocols like plain FTP or SCP.
- Data Staging and Persistence: Stage files in highly available, durable, encrypted Object Storage, with lifecycle policies for retention and archiving based on business and compliance needs.
- File Naming and Metadata: Agree conventions for file names, directories and metadata so internal and external systems can process and audit files automatically.
- Monitoring and Auditing: Log all transfer activity (transfers, failures, authentication attempts, file access) on the gateway and Object Storage, and feed it to a central observability platform.
- Error Handling and Retries: Use error handling, automatic retries with backoff and alerts on failed transfers.
- Transformation and Orchestration: Hand complex transformations, protocol conversions and multi-step workflows to dedicated Middleware Services or Data Integration Platforms, not the transfer gateway.
- Security Hardening: Regularly audit and harden the gateway: disable unused protocols, enforce strong cipher suites and rotate credentials.
Security Controls
- Perimeter Security: Place the Managed Secure File Transfer Gateway in a Public Subnet (Perimeter) behind Edge Protection (WAF) to defend against common web vulnerabilities and brute-force attacks.
- Transport Security: Encrypt all data in transit: SFTP runs over SSH, and HTTPS and FTPS (FTP over TLS) must use Transport Layer Security (TLS 1.2 or higher). Prohibit legacy or unencrypted protocols like FTP.
- Authentication & Authorisation:
- For system-to-system integrations, use SSH Key Pairs (with strong passphrases) or API Key authentication for SFTP/HTTPS transfers.
- For external users, authenticate through an identity provider with OAuth 2.0/OpenID Connect (OIDC), then apply granular access control lists (ACLs) to file directories or object storage buckets.
- Data at Rest Security: Encrypt all files staged in the gateway or Object Storage with platform-managed or customer-managed keys.
- Malware and Vulnerability Scanning: Automatically scan every incoming file for malware and other malicious content on arrival, before internal workloads process it.
- Data Loss Prevention (DLP): Where compliance requires it, configure the gateway and egress protection to inspect outbound files and prevent data exfiltration.
- Data Segregation: Never store sensitive or processed files in the Public Subnet (Perimeter); use durable object storage in private subnets.