frontend / frontend system design / 07_file_uploader_with_resume.md

Design: File Uploader with Resume

7 min read source

Design: File Uploader with Resume

TL;DR

Drag-and-drop multi-file uploader that survives flaky networks, large files, and tab refresh. The senior topics: direct-to-storage (presigned URLs, not your server), chunked / multipart for resilience, resumable protocol (S3 multipart or tus), concurrency cap, per-file progress + cancel, integrity verification, and the security perimeter in the presign step.

Requirements to clarify

  • Max file size. 10MB photos? 5GB videos? Drives chunking strategy.
  • Connection profile. Mobile / hotel Wi-Fi (flaky) vs office (stable)?
  • File count per session. 1 at a time, or “drop 500 photos”?
  • Resume across tab refresh / day later? Yes → need persistent state and server-side upload sessions.
  • Storage destination. S3 (multipart upload API) / GCS / self-hosted (tus)?
  • Post-upload processing. Trigger Lambda for thumbnails? Notify the user when done?
  • Privacy / signed download. Public after upload, or signed-URL only?

Architecture

[Client]
   │ 1. Validate file (client-side)
   ├──▶ [API] /uploads/init   (auth, validate, generate UploadId + presigned URLs per chunk)
   │         │
   │         └──▶ [Storage backend creates multipart upload session]
   │
   │ 2. PUT each chunk to presigned URL (in parallel, concurrency cap)
   ├──▶ [Storage (S3 / GCS / tus server)]
   │
   │ 3. Notify completion with chunk ETags
   ├──▶ [API] /uploads/complete  (server calls CompleteMultipartUpload, optionally triggers post-processing)
   │
   └──▶ User sees confirmation

Server is never on the data path. The bytes go directly from browser to S3.

See ../11_apis_data_fetching/09_file_uploads.md for the protocol details (multipart, presigned, tus).

API contract

POST /api/uploads/init
  { "filename": "video.mp4", "size": 5_368_709_120, "contentType": "video/mp4", "checksum": "sha256:..." }
→ {
    "uploadId": "u_abc",
    "key": "uploads/u_ada/2026/05/video.mp4",
    "partSize": 8_388_608,                              // 8 MB
    "parts": [
      { "partNumber": 1, "url": "https://s3...?...&partNumber=1", "expiresAt": "..." },
      { "partNumber": 2, "url": "...", "expiresAt": "..." },
      ...
    ]
  }

PUT to each part URL → response includes ETag header (must store)

POST /api/uploads/:uploadId/complete
  { "parts": [{ "partNumber": 1, "etag": "..." }, ...] }
→ { "fileId": "f_123", "url": "https://cdn/.../video.mp4" }

POST /api/uploads/:uploadId/abort
→ {}     // cancel the S3 multipart upload, clean up

The server validates size and contentType against an allow-list before signing — that’s the security boundary, not the client.

Client state per file

type UploadFile = {
  id: string;                        // local UUID
  file: File;
  status: "queued" | "uploading" | "paused" | "done" | "failed" | "cancelled";
  uploadId?: string;                 // server-issued
  key?: string;
  parts: PartState[];                // per-chunk state
  progress: number;                  // 0..1 derived from parts
  error?: string;
};

type PartState = {
  partNumber: number;
  url: string;
  expiresAt: string;
  etag?: string;
  status: "pending" | "uploading" | "done" | "failed";
  attempts: number;
};

Persisted to IndexedDB so a refresh/quit can resume. The File reference can’t be serialized — the user has to re-pick the file to resume (browsers don’t allow durable file handles, except via the experimental File System Access API). The recovered IDB state lets the UI say “you had a video at 60% — re-add it to resume.”

Chunking strategy

  • Min part size (S3): 5 MiB except the last part.
  • Max parts (S3): 10,000.
  • A 50 GB file at 5 MiB parts = 10K parts (boundary). Pick a part size proportional to file size: partSize = max(8 MiB, ceil(size / 9500)).
  • Larger parts = fewer requests, larger blast radius on a single failure. 8-16 MiB is the sweet spot for most files.

Concurrency cap

Upload N parts in parallel; cap at 4-6.

async function uploadParts(file: UploadFile, concurrency = 4) {
  const queue = file.parts.filter(p => p.status === "pending");
  const workers = Array.from({ length: concurrency }, async function worker() {
    while (queue.length) {
      const part = queue.shift()!;
      try { await uploadOnePart(file, part); }
      catch (e) { part.status = "failed"; part.attempts++; /* requeue per retry policy */ }
    }
  });
  await Promise.all(workers);
}

Why cap: 100 parallel PUTs saturate the browser’s 6-per-origin (HTTP/1.1) limit, exhaust bandwidth, increase per-part latency, and trigger S3 throttling. 4-6 is the published sweet spot.

Progress UX

  • Per-part progress via XMLHttpRequest.upload.onprogress (not fetch, which is unreliable — see ../11_apis_data_fetching/09_file_uploads.md).
  • Per-file progress = sum of part bytes uploaded / total bytes.
  • Aggregate progress = sum across all files in queue.
  • ETA = remaining bytes / recent throughput (compute over a rolling window so a momentary stall doesn’t lie).
  • Pause / Resume / Cancel per file (AbortController for in-flight requests; queue manipulation for queued).

Failure modes and recovery

  • Part fails (network error, 5xx) → retry that part with exponential backoff + jitter (cap at e.g. 5 attempts). Don’t restart the whole file. See ../11_apis_data_fetching/06_retries_backoff_idempotency.md.
  • Presigned URL expired mid-upload → ask the server for a fresh URL for that part (POST /uploads/:id/parts/:n/resign) and retry.
  • Tab refresh → IDB has the upload state; user re-picks the file; client compares file size + (optionally) hash; if it matches, resume from parts where status !== "done".
  • Network drop → all in-flight parts fail; on reconnect, retry the failed parts.
  • Full upload session expires (some servers expire multipart uploads after N hours) → restart with a fresh init.
  • Abort → client calls /uploads/:id/abort so S3 doesn’t keep paying for the orphan parts. Set a bucket lifecycle policy as a backstop: “Abort incomplete multipart uploads after 7 days.”

Integrity verification

  • Client computes the file SHA-256 (using crypto.subtle.digest on the File’s ArrayBuffer — stream it for big files via ReadableStream) and sends with init.
  • Server can verify by checking x-amz-meta-checksum or by retrieving and hashing post-upload. For mission-critical files, this catches corruption (truncated, bit-flipped) before the upload is marked done.
  • Per-part checksums — S3 supports Content-MD5 per part; client computes and sends with each PUT. S3 rejects on mismatch.

Validation and security

  • Client-side: extension allow-list, max size, magic-byte sniff for the file’s actual type (don’t trust extension).
  • Server-side (presign): re-validate size, content type allow-list, scope key prefix per user (uploads/${userId}/...), short URL expiry.
  • Server-side (complete): actually fetch the object’s HEAD and verify the stored size + content type match what was promised. The client can lie at PUT time; this is the last checkpoint.
  • Post-upload scan: anti-malware scan via S3 event → Lambda → ClamAV or commercial service before exposing.

See ../17_security/ for the broader file-upload security checklist.

Drag-and-drop UX

  • Drop zone visible always; highlights on dragenter.
  • File picker fallback (<input type="file" multiple>) for keyboard / mobile.
  • Reject files that fail client-side validation with an inline error (don’t fail silently).
  • Show a queue with each file’s state inline.
  • Don’t onDragOver without preventDefault() — the default behavior of dropping a file onto a page is to navigate to it.
<div
  onDragOver={(e) => { e.preventDefault(); setHover(true); }}
  onDragLeave={() => setHover(false)}
  onDrop={(e) => { e.preventDefault(); setHover(false); enqueue([...e.dataTransfer.files]); }}
>
  Drop files here or <input type="file" multiple onChange={(e) => enqueue([...e.target.files!])} />
</div>

When to use a library

  • Uppy — full uploader UI with provider plugins (Google Drive, Dropbox), tus, S3 multipart, progress, retry. The default for “I need this in production tomorrow.”
  • tus-js-client — protocol implementation; bring your own UI.
  • @aws-sdk/lib-storage — S3 multipart upload in the browser; lower level than Uppy.

A senior answer mentions one of these by name and explains why hand-rolling is rarely worth it.

Telemetry

  • p99 upload latency per MB.
  • Retry rate per part.
  • Resume rate (% of uploads that experienced a resume).
  • Abort rate (% of users who cancel mid-upload).
  • Avg parts per file (ties to chunk-size choice).

What a senior is expected to say

  • “Direct-to-S3 with presigned URLs; server never on the data path. Multipart upload for anything >100MB or any flaky network.”
  • “Per-part retries with exponential backoff. URL re-signing for parts whose presigned URL expires mid-upload.”
  • “Concurrency cap at 4-6 parallel parts — beyond that you fight browser connection limits and S3 throttling, with no benefit.”
  • “Resumability requires persisted upload-session state in IndexedDB. File objects can’t be re-attached durably; the user re-picks the file, we verify size+hash, resume from incomplete parts.”
  • “Security boundary is the presign endpoint — that’s where size, type, and key prefix are validated. Plus a post-upload verification of stored metadata.”
  • “For production, I’d use Uppy or tus-js-client unless we have an unusual reason to hand-roll.”

Cross-references

Further reading

Previous
Design: Document Comments (Google-Docs Style)