Define an exchange contract, not just connection details
An SFTP integration often begins with a hostname, credentials, and a directory path. That is enough to establish a connection, but not enough to operate a dependable business process. Both parties need to agree on who produces each file, when delivery is considered complete, how quickly it should be processed, who owns failures, and how long source files remain available. If those decisions live only in email threads, an innocent scheduling or naming change can become a difficult production incident.
The contract should cover filename conventions, directory purpose, encoding, delimiters, quoting, line endings, time zones, null representation, field types, schema versions, and control totals. CSV is especially deceptive: embedded commas, leading zeros, spreadsheet conversions, and mixed encodings regularly cause trouble. Fixed-width files require byte-level definitions rather than visual column positions. For sensitive content, decide whether transport encryption is sufficient or whether files also require PGP encryption. Key ownership, expiry, and rotation must be designed before the first emergency renewal.
Use atomic delivery to avoid consuming partial files
A receiver should never assume that a visible final filename is complete. A polling job can discover a large file while it is still being uploaded and begin parsing incomplete data. A safer pattern is to upload under a temporary extension or into a staging directory, then rename or move the file into the inbound directory after the transfer finishes. A rename within the same SFTP filesystem is commonly close to atomic, but the behavior should still be verified against the actual server.
If the sender cannot rename files, it can publish a separate completion marker. Another fallback is to wait until size and modification time remain unchanged across multiple observations. That approach introduces a trade-off: a short stability window can misclassify a paused upload, while a long window adds latency. Regardless of the publication method, validate file size, checksum, or signature. A successful SFTP session proves that bytes were transferred; it does not prove that the intended business file is complete and correct.
- Upload: Write to a temporary name and publish the final name only after completion.
- Validate: Check schema version, required fields, encoding, record totals, and checksum.
- Process: Send only validated data to the ERP, CRM, or data platform.
- Archive: Preserve the original input and processing result; move invalid files to quarantine.
Design explicitly for duplicates and interrupted processing
Assume every file may be observed more than once. A partner may resend after a timeout, scheduled jobs may overlap, or a worker may restart after writing downstream data but before marking the file complete. A design that uses only the current contents of an inbound directory as its memory can easily create duplicate orders, inventory movements, or customer records.
Maintain a processing ledger containing the source, filename, size, content hash, contract version, first-seen time, status, and outcome. Useful states include discovered, stable, validated, processing, completed, failed, and quarantined. Do not use the filename as the only identity because senders sometimes reuse names. A content hash identifies an exact resend, but it may not be sufficient when identical content is legitimately exchanged on different dates. In that case, combine it with a sender-provided batch ID or exchange ID.
Idempotency must continue into downstream systems. Ideally, business writes and the ledger update share one transaction boundary. When processing crosses databases or APIs, use stable idempotency keys, record item-level outcomes, and implement a recoverable state machine. If a file partly succeeds, the recovery policy must be explicit: roll back, skip confirmed items, or issue compensating actions. Reprocessing the whole file blindly is rarely a safe default.
Coordinate retries, alerts, and controlled replay
Retrying every error is not resilience. Network interruption, temporary DNS failure, and server throttling are reasonable candidates for bounded exponential backoff. A malformed file, unsupported schema version, or missing required field will not heal with time and should be quarantined promptly. Revoked permissions, changed host keys, and exhausted storage also require actionable alerts rather than endless background attempts.
Operational visibility should answer both engineering and business questions: Did the expected file arrive on time? Which state is it in? How many retries have occurred? When was the last successful exchange? Which downstream records were committed? Logs should carry an exchange ID and file fingerprint without exposing credentials or sensitive payloads. Alerts should include the source, filename, failure category, current state, and likely next action so responders do not have to reconstruct the incident from several systems.
Replay should be a supported operation, not an improvised file move over SSH. A command or administrative workflow should identify the exact file, scope, and reason for replay, apply the same idempotency controls as normal processing, and record the operator and result. Retain original files and validation evidence according to governance requirements, but do not treat the SFTP directory itself as a permanent archive.
Test failure scenarios before production
Integration testing should go well beyond one valid sample. Exercise interrupted uploads, truncated and empty files, reused filenames, duplicate content, late arrival, out-of-order batches, unsupported versions, encoding errors, downstream timeouts, and worker restarts between processing and acknowledgement. Security tests should cover SSH host-key pinning, least-privilege accounts, network allowlists, private-key storage, and rotation. A changed host key should stop the connection for verification, not be accepted automatically.
Finally, assign business and technical owners for every exchange, and define operating hours, delay tolerance, cutoff rules, and incident procedures. When file exchange supports a critical workflow, its contract, ledger, monitoring, and replay tooling are production integration components, not optional conveniences. If an integration team is involved, evaluate it on these operational mechanisms rather than on whether it can merely establish an SFTP connection.