Integration Secure File Transfer (Internal)
Problem
Backend workloads that your teams run in private subnets hand files to each other for batch processing, data sync and event-driven work, so the controls are workload identity (IAM roles, service accounts or SSH keys) and encryption in transit and at rest. When each team moves files its own way, integrity checks get skipped, the audit trail breaks, and files can be intercepted or lost even though they never leave the private network.
Solution
Keep the exchange private: a Managed File Transfer Service and highly available Object Storage, reached through private endpoints, with each workload authenticating by IAM role, service account or SSH key under least privilege. It supports push (the source writes to Object Storage, triggering delivery or processing), pull (the destination reads from it) and direct secure channels (SFTP over SSH, or FTPS, which is FTP over TLS), with integrity checks and traceability.
Cloud Paradigm
- Managed Services (for file transfer orchestration and object storage)
- Event-Driven Architecture (e.g., object creation triggers downstream processing)
- Data as a Service (DaaS) principles for internal data distribution
- Infrastructure as Code (IaC) for service configuration
- Zero Trust principles for access to data and transfer components
Solution Flow
Data Transfer Flow (Push-based via Object Storage):
- Source Backend Workload: The internal microservice or batch process generates or prepares a file for transfer.
- Secure Ingestion to Object Storage: The source uploads the file to a designated bucket in a highly available Object Storage service, usually through authenticated APIs over HTTPS.
- Event Notification (Optional): The upload can emit an event notification (e.g., an object creation event) that triggers downstream processing or transfer orchestration.
- Managed File Transfer Service / Processing Workload: A Managed File Transfer Service or an event-driven processing workload (e.g., a serverless function) detects the file, validates it, scans it for malware and may run basic integrity checks.
- Secure Delivery to Destination: The service delivers the file to the destination workload's designated location (e.g., another Object Storage bucket, a mounted file system or a processing queue) under the configured access controls.
- Destination Backend Workload: The consuming microservice or batch process retrieves the file from its secure endpoint, processes it and confirms receipt.
Data Transfer Flow (Direct Secure Channel):
- Source Backend Workload: The internal microservice or batch process connects to the Managed File Transfer Service (e.g., an SFTP/FTPS gateway endpoint) in a Private Subnet.
- Managed File Transfer Service: The service authenticates the source with secure credentials (e.g., SSH keys, IAM roles) and receives the file over an encrypted channel (SSH for SFTP, TLS for FTPS), scanning for malware and checking integrity during or just after ingestion.
- Secure Delivery to Destination: The service delivers the file to the destination's designated endpoint: another secure gateway, a specific storage location or the consuming application directly.
- Destination Backend Workload: The consuming microservice or batch process retrieves and processes the file.
When to Use
- Microservices or batch jobs in Private Subnets exchange discrete files (CSV, Parquet, Avro) for scheduled processing or data synchronisation.
- Workflows benefit from event-driven triggers, where an object-creation event starts validation, scanning or downstream delivery.
- Legacy or partner systems need a managed SFTP/FTPS endpoint, but you want endpoints on private networking with IAM-backed credentials.
- Compliance requires traceability, malware scanning and integrity checks on every file passed between workloads.
- You need decoupled producers and consumers, with Object Storage as a durable, auditable buffer between mismatched processing rates.
When NOT to Use
- You need low-latency, record-level streaming. Use an event streaming or message queue pattern instead.
- Payloads are small synchronous request/response calls. A direct API or service-mesh call is simpler.
- Heavy transformation, enrichment or protocol translation is the main need. Use a dedicated ETL/ELT or data integration pipeline.
- Files must cross organisational boundaries over the public internet, with external partner onboarding. A hardened external-facing MFT/DMZ pattern fits better.
- You need transactional consistency across multiple datasets. A database or saga-based approach beats file drops.
Trade-offs
- Durable, auditable exchange via Object Storage vs storage cost and the work of managing retention, archiving and secure deletion.
- Decoupled producers and consumers vs eventual-consistency latency and the need for idempotent receivers that handle duplicate deliveries.
- Central scanning, validation and traceability vs running and patching a Managed File Transfer Service or event-driven functions.
- Standard secure protocols (SFTP/FTPS/HTTPS) vs the friction of moving legacy workloads off insecure FTP and managing keys and credentials.
- Reliable retries and dead-letter handling vs building error handling, monitoring and alerting around every transfer stage.
Illustrative Scenario
A hypothetical scenario showing how the pattern applies. Not a client case study.
In a hypothetical broadcaster, finished program files move between post-production render microservices and the playout scheduling platform, both isolated in Private Subnets. Neither side connects to the other directly; the bucket sits between them. When an encoding batch finishes, the render workload uploads a Parquet manifest and its media package to a designated Object Storage bucket over HTTPS. The object-creation event triggers a serverless function. It validates the asset metadata against the schema, scans for malware and checks integrity, and only then does the Managed File Transfer Service deliver the package to the playout team's ingestion bucket. Failed deliveries go to a dead-letter queue with exponential-backoff retries, and idempotent receivers absorb any duplicate drops. Transfer status, latency and file-size metrics feed central alerting, so operations can catch a stalled transfer before its scheduled broadcast slot. Retention policies move aired content to lower-cost storage tiers, and full traceability is kept throughout the whole process.
Additional Details
- Protocols: Prefer SFTP (over SSH), FTPS (FTP over TLS) or HTTPS for direct API calls to Object Storage. Avoid legacy, insecure protocols like FTP.
- File Formats & Schema: Standardise formats (e.g., JSON, XML, CSV, Parquet, Avro), define clear schemas and validate them during ingestion or processing.
- Orchestration & Scheduling: Prefer event-driven triggers (e.g., file arrival starts processing). For batch work, configure reliable scheduling with retries and dependencies.
- Data Transformation: Leave complex transformation, enrichment or protocol translation to Data Integration services or ETL/ELT pipelines, not the core file transfer component.
- Error Handling & Retries: Use error handling, automatic retries with exponential backoff, and dead-letter queues for failed transfers or processing.
- Monitoring & Alerting: Monitor transfer status, latency, file sizes and error rates, with central alerting so operations teams hear about failures or anomalies.
- Lifecycle Management & Retention: Define retention periods, archiving to lower-cost storage tiers and secure deletion for files in Object Storage.
- Idempotency: Make receiving workloads idempotent so duplicate deliveries do no harm, especially in distributed or event-driven architectures.
Security Controls
- Transport Security: Encrypt all data in transit: use SFTP (over SSH) or FTPS (FTP over TLS) for file transfers, and require Transport Layer Security (TLS 1.2 or higher) for FTPS and for HTTPS object storage API interactions.
- Data at Rest Encryption: Encrypt all files in Object Storage with platform-managed or customer-managed encryption keys.
- Authentication & Authorisation:
- Authenticate workloads to the transfer service and object storage with managed Identity and Access Management (IAM) roles, service accounts or SSH keys (kept in a secrets management service).
- Apply least privilege: grant only the permissions each transfer operation and bucket needs.
- Vulnerability Scanning & Malware Detection: Automatically scan every incoming file for malware, viruses and other threats on ingestion into Object Storage or during transfer.
- Network Isolation: Keep transfer components and Object Storage endpoints in Private Subnets, reachable only by authorised workloads through private endpoints or secure gateways.
- Audit & Traceability: Log every transfer's source, destination, file name, size, timestamps and status to a central logging platform as immutable records.
- Data Integrity: Use checksums (e.g., MD5, SHA256) during transfer and storage to verify integrity and detect corruption.