Integration Secure File Transfer (Cloud)
Problem
Partners, customers and third-party systems you do not control need to exchange large files or batch data with your cloud environment, over the public internet or a dedicated interconnect, so every connection must enter through a WAF-protected perimeter and authenticate with SSH keys or OAuth 2.0. Their capabilities and security posture differ from yours and the data is batch by nature, so direct integration is often impractical, yet every transfer must be secure, reliable and auditable.
Solution
Run a Managed File Transfer (MFT) Service or Secure File Transfer Gateway in the Public Subnet (Perimeter), behind a WAF, as the only point outsiders reach. It authenticates senders, scans files and moves them straight to encrypted Object Storage in a Private Subnet (Workloads), and sends outbound files the other way, with every transfer checked and traced.
Cloud Paradigm
- Managed File Transfer as a Service
- Data Ingestion/Egress Patterns
- Serverless Data Processing and Orchestration
- Object Storage for scalable and secure data staging
- Hybrid Connectivity via Dedicated Network Interconnects
- Automation of Data Transfer Workflows
- Data Governance and Auditing
Solution Flow
Data Ingestion Flow (External to Internal):
- External Sender: A third-party system or partner starts a secure transfer (e.g., via SFTP, FTPS or an API-driven upload) to the Managed File Transfer Gateway.
- Web Application Firewall (WAF) / Network Edge: A WAF inspects the connection for malicious activity and unauthorised access attempts before it reaches the MFT Gateway.
- Managed File Transfer Gateway: In the Public Subnet (Perimeter), the gateway authenticates the sender (e.g., SSH key, API key), terminates the encrypted session (SSH for SFTP, TLS for FTPS or HTTPS), scans the file for malware and runs the transfer. It does not persistently store the file.
- Temporary Staging (Private Object Storage): The gateway moves the file to designated Object Storage in a Private Subnet (Workloads), where it is encrypted at rest. A successful upload triggers a notification (e.g., message queue, event).
- Backend Workload: An internal processing service or microservice (e.g., ETL pipeline, serverless function) consumes the file, applies business logic or transformations, or loads it into target systems.
Data Egress Flow (Internal to External):
- Backend Workload: An internal application or processing service prepares outbound files and places them in a designated Object Storage bucket in a Private Subnet (Workloads).
- Managed File Transfer Gateway Trigger: A notification (e.g., message queue, event) from Object Storage or the application triggers the Managed File Transfer Gateway.
- Managed File Transfer Gateway: The gateway retrieves the file, runs final security checks (e.g., DLP, malware scan) and sends it to the external receiver using predefined credentials and protocols.
- External Receiver: The third-party system or partner receives the file from the MFT Gateway.
When to Use
- You exchange large files or batch data with external partners whose capabilities, protocols or security posture you cannot dictate.
- Regulation or contracts demand full auditability, data lineage and traceability for every file movement.
- Partners rely on established protocols like SFTP, FTPS or PGP-encrypted drops rather than real-time APIs.
- You need a hardened perimeter broker that authenticates senders, scans for malware and terminates encrypted sessions before files reach internal networks.
- Exchange is periodic or scheduled (nightly reconciliations, payroll runs, EDI feeds), not transactional.
When NOT to Use
- You need real-time or low-latency exchange; use event streaming or synchronous API integration instead.
- Both parties run modern systems that can integrate directly over REST/gRPC APIs with mutual TLS.
- Payloads are small, discrete records better suited to a message queue or webhook pattern.
- Data must be transformed and correlated in flight across many sources; an ESB or iPaaS flow fits better.
- Transfers are purely internal within one trust boundary, where a shared object store or database suffices.
Trade-offs
- Strong perimeter isolation and malware scanning vs running and patching a hardened MFT gateway in the public subnet.
- Broad protocol compatibility with varied partners vs the batch latency of file-based exchange.
- Full audit trails and data lineage vs extra storage governance, retention management and compliance tooling costs.
- Transfer decoupled from internal processing via staged object storage vs eventual-consistency complexity and post-transfer orchestration.
- Centralised credential and key management vs onboarding, rotating and revoking per-partner credentials at scale.
Illustrative Scenario
A hypothetical scenario showing how the pattern applies. Not a client case study.
Imagine a state tax authority that receives nightly wage and withholding filings from thousands of employers and payroll bureaus still tied to SFTP and PGP-encrypted drops. Each submitter authenticates to the Managed File Transfer Gateway in the perimeter subnet with its own SSH key, so access can be revoked per submitter. A WAF inspects the connection first. The gateway then terminates the SSH session, scans each file for malware and streams it, without keeping a local copy, into encrypted object storage in the private workload subnet. An upload event triggers a serverless ETL pipeline that validates the schema, converts the records to Parquet and loads them into the revenue data warehouse. For outbound refund confirmations and assessment notices, internal systems place files in a designated bucket. An event then prompts the gateway to run a DLP check and deliver each file over SFTP. Every transfer is logged with full lineage to meet statutory audit and public-records obligations.
Additional Details
- Supported Protocols & Formats: Support standard secure protocols such as SFTP and FTPS, plus HTTPS for API-driven transfers where needed. Agree file formats with partners, preferring structured formats (CSV, Parquet, JSON, XML) to unstructured text.
- Automation & Orchestration: Use workflow orchestration tools or serverless functions to automate post-transfer processing (e.g., validation, loading into data warehouses, triggering downstream systems) and pre-transfer preparation of outbound files.
- Data Governance: Set clear retention policies for files in Object Storage, and keep lineage and audit trails for every transfer.
- Error Handling & Retries: Handle errors, retry transient failures automatically and alert on persistent issues.
- Scalability & High Availability: Design the MFT service and Object Storage to scale horizontally and stay highly available as transfer volumes change.
- Observability: Log, measure (transfer duration, file size, success and failure rates) and alert on the gateway and Object Storage, feeding central monitoring for end-to-end visibility.
Security Controls
- Perimeter Security: Deploy the Managed File Transfer Gateway in a Public Subnet (Perimeter) behind a Web Application Firewall (WAF) to block common network attacks and unauthorised access.
- Transport Security: Encrypt all data in transit with SFTP (over SSH), FTPS (FTP over TLS) or HTTPS, enforcing Transport Layer Security (TLS 1.2 or higher) wherever TLS is used. For high-volume or sensitive transfers, consider Dedicated Network Interconnects instead of the Public Internet.
- Authentication & Authorisation:
- For programmatic access to MFT APIs, use OAuth 2.0 (Client Credentials Grant) or Managed Identities.
- For client access (e.g., SFTP), use strong SSH keys or secure credentials, with multi-factor authentication (MFA) where possible.
- Use Role-Based Access Control (RBAC) for MFT platform users and for access to staged files in Object Storage.
- Data at Rest Encryption: Encrypt all staged files in Object Storage with Customer-Managed Keys (CMK) or platform-managed keys.
- Data Integrity: Verify integrity with checksums or hashes during and after transfer.
- Malware and Vulnerability Scanning: Automatically scan every inbound and outbound file for malware, viruses and other threats on ingestion and before egress.
- Storage Residency: Files must never be persistently stored in the Public Subnet (Perimeter). Move ingested files immediately to Object Storage in a Private Subnet (Workloads).
- Auditing & Logging: Log all transfers, access attempts and administrative actions, and send the logs to a Security Information and Event Management (SIEM) system for real-time monitoring.
- Data Loss Prevention (DLP): Inspect outbound file content for sensitive data with DLP policies to prevent unauthorised disclosure.