Walkthrough: Designing a File Storage & Sync Service

The problem

You’re asked to design a file storage and sync service — Dropbox, Google Drive, OneDrive. The interviewer says something like: “Users install a client on several devices. Files they drop in a folder upload automatically, sync to their other devices, can be shared with other users, and keep version history.”

Sounds like upload-to-S3 with a database. It isn’t. The trap is designing a file transfer service when the question is a file sync system: the hard parts are the split between content and metadata, chunking (which buys resumability, dedup, and delta sync in one move), and what happens when two offline devices edit the same file. The bytes are the easy half; the bookkeeping is the design.

Below is how I’d walk through this, start to finish.

1. Requirements

First 3–5 minutes. Questions I’d ask to pin down what we’re building:

Functional requirements

• R1. Upload and download files (up to 10 GB) from any device.
• R2. Automatic sync — an edit on one device appears on the user’s other devices within seconds, transferring only what changed.
• R3. Sharing — folders/files shared with other users, with permissions.
• R4. Version history — restore previous versions; recover deleted files.
• R5. Offline edits — devices reconcile when they reconnect; concurrent edits must never silently lose either side.

Scope

Explicitly out of scope:

• Real-time collaborative editing (CRDT/OT merging is the collaborative-editor question; this system’s unit of conflict is the file)
• Full-text search inside file contents (a search pipeline fed by this system)
• Client filesystem-watcher internals (I’ll treat “client detects a change” as given)
• Malware scanning, legal holds, enterprise DLP (filters on the upload path; they don’t shape the architecture)

Non-functional requirements

• Scale. Users, files per user, average file size — this sets both the exabyte question and the metadata-rows question.
• Durability vs availability split. File content loss is unacceptable (11 nines territory); brief metadata unavailability is survivable.
• Sync latency. Seconds, not milliseconds — worth confirming because it rules out heroic real-time machinery.
• Edit patterns. Are big files edited in place (a 2 GB video project) or replaced wholesale? Decides how much delta sync is worth.

Say the interviewer confirms: 500M registered users, 100M DAU, ~10K files per user average (trillions of file records), average file ~2 MB with a long tail to 10 GB, edit-to-sync p95 under 10 seconds, 30-day version history, content durability 99.999999999%, and yes — concurrent offline edits happen daily at this scale.

2. Capacity estimate

• Content: 500M users × ~4 GB stored average ≈ 2 exabytes logical. Even at 30% dedup + compression savings and with replication/erasure overhead, this is exabyte-class physical storage — the per-byte economics (dedup, cold tiering, erasure coding instead of 3× replication) are first-order design inputs, not ops details.
• Metadata: 500M users × 10K files ≈ 5 trillion file records — far past one database; the metadata tier is itself a sharded distributed system.
• Traffic: 100M DAU × ~20 sync operations/day ≈ ~25K metadata ops/sec average, peaks 10×. Content bandwidth: ~1B file transfers/day × 2 MB ≈ ~25 GB/sec average egress+ingress — CDN-shaped for hot shared content.

I’d say out loud: “Two different systems are hiding in the numbers. An exabyte content store where immutability and cost dominate, and a trillion-row metadata store where transactional consistency dominates. They fail differently, scale differently, and should be designed separately — that’s the architecture in one sentence.”

3. High-level architecture

Three logical subsystems:

1. Metadata service — the namespace: files, folders, versions, shares, device sync cursors. Transactional, sharded.
2. Content service — chunk upload/download against object storage; content-addressed, immutable, CDN-fronted for hot reads.
3. Sync & notification — tells devices “your view is stale”; devices then pull a metadata diff and fetch exactly the chunks they lack.

flowchart TD
  Client[Device client] --> Meta[Metadata service]
  Client --> Content[Content service]
  Content --> ObjStore[(Object storage · chunks)]
  Meta --> MetaDB[(Metadata DB)]
  Notify[Notification service] -.->|"you're stale"| Client

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class ObjStore,MetaDB store

The metadata service decides the system’s character: every sync, share, and restore is a metadata transaction; content is just bulk bytes the metadata points at.

4. Architecture evolution

Each level keeps the prior contract: a Level 3 client still just uploads and downloads — it simply transfers chunks instead of files.

The design in this walkthrough is Level 3–4: chunked, content-addressed, sharded metadata.

I’d say out loud: “Level 2 — whole files in S3 — is a fine v1 and I’d say so. Chunking earns its complexity the moment files are large, edits are small, or storage cost matters. Our requirements hit all three.”

Target architecture (Levels 3–4)

Every section that follows explains one part of this diagram:

flowchart TD
  Client[Device client] --> Meta[Metadata service]
  Meta --> MetaDB[(Metadata DB · sharded by namespace)]
  Client --> Content[Content service]
  Content --> ObjStore[(Object storage · chunks)]
  CDN[CDN] --> ObjStore
  Client -.->|hot shared reads| CDN

  Meta -->|commit event| Notify[Notification service]
  Notify -.->|long-poll push| Client2[User's other devices]
  Client2 --> Meta

  GC[Chunk GC] -.-> MetaDB
  GC -.-> ObjStore

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class MetaDB,ObjStore store

5. Core design choice: how content is addressed

Three options:

(a) Whole-file blobs. file_id → blob. Simple, correct, and the right v1. Fails our requirements three ways: a 1-line edit to a 2 GB file re-uploads 2 GB (R2’s “only what changed”), a dropped connection restarts from byte zero (R1 at 10 GB), and version history stores near-identical blobs (R4 × cost).

(b) Fixed-size chunks, content-addressed. Split at fixed boundaries (4 MB). Each chunk’s ID is its SHA-256. A file version is a manifest: an ordered list of chunk hashes. Dropbox’s classic design.

(c) Content-defined chunking (CDC). Boundaries chosen by a rolling hash over content, so inserting bytes early in a file doesn’t shift every subsequent chunk boundary — better dedup for insert-heavy edits, at the cost of variable chunk bookkeeping and CPU on every client.

Choice: (b) — fixed 4 MB chunks — with (c) named as a targeted upgrade. The workload-specific justification: the corpus is dominated by media and documents that are replaced or appended, not byte-shifted; fixed chunking captures append and tail-edit deltas, dedups identical files perfectly, and keeps the chunk math trivial on a billion devices. CDC’s win case (insertions in huge mutable files) is real but narrow — the rsync lineage — and can ship later for file types that earn it.

Tradeoff matrix

Architecture decisions

Core data structures

6. Write path: upload and commit [R1, R2, R4]

flowchart TD
  Client[Device client] -->|"1 · hash 4MB chunks locally"| Client
  Client -->|"2 · which of these hashes do you have?"| Meta[Metadata service]
  Meta -->|"3 · missing: 2 of 5"| Client
  Client -->|"4 · upload missing chunks"| Content[Content service]
  Content --> ObjStore[(Object storage)]
  Client -->|"5 · commit manifest"| Meta
  Meta --> MetaDB[(Metadata DB)]
  Meta -->|"6 · event"| Notify[Notification service]

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class ObjStore,MetaDB store

Step by step, for a 20 MB file edit:

1. The client detects the change, splits the file into five 4 MB chunks, and hashes each locally. Hashing on the client is the design’s quiet load-shifter: a billion devices do the CPU work, and the hash is the dedup query.
2. Client asks the metadata service which hashes are new. Three of five match the previous version (the edit touched the file’s tail); one matches a chunk another user uploaded last year (global dedup); one is genuinely new.
3. Client uploads only the missing chunk to the content service, which verifies the hash on receipt (content addressing makes integrity checking free) and writes to object storage.
4. A dropped connection here costs nothing: re-running step 2 shows what arrived; upload resumes at the missing chunk (R1’s resumability).
5. Client commits: new version of file F = manifest [h1..h5], parent version v17. The metadata transaction checks the parent is still current (the optimistic-concurrency check that catches conflicts, §8), inserts the version, bumps the namespace sequence number.
6. The commit emits an event; the notification service wakes the user’s other devices and any shared-folder members (§9’s fan-out).

Why content-first commit ordering is load-bearing: if the client dies after step 4 but before 5, object storage holds orphan chunks — harmless, swept by GC later. Reverse the order and a crash leaves a committed manifest pointing at chunks that never arrived: a file that exists and can never be downloaded. Orphans are garbage; dangling pointers are corruption. Always choose the failure that’s garbage.

7. Read path: sync and download [R2]

flowchart TD
  Notify[Notification service] -.->|"1 · wake: namespace changed"| Client2[Other device]
  Client2 -->|"2 · delta since cursor 4711"| Meta[Metadata service]
  Meta -->|"3 · changed manifests"| Client2
  Client2 -->|"4 · fetch chunks not held locally"| Content[Content service]
  Content --> ObjStore[(Object storage)]
  Client2 -.->|hot shared chunks| CDN[CDN]

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class ObjStore store

Step by step:

1. The notification service pokes connected devices (long-poll/SSE — the real-time connection patterns reference covers the transport choice). The poke carries no data — it’s purely “you’re stale.”
2. The device pulls delta(namespace, cursor=4711): every metadata change since the sequence number it last saw. Cursors make sync stateless on the server and identical for a device that slept one second or one month — reconnection is just a bigger delta (R5’s offline half).
3. The delta lists new manifests. The client diffs each against chunks it already holds (its local store is also content-addressed) and computes the minimal fetch set — delta sync in reverse.
4. Chunk downloads hit the CDN for hot content (a file shared to a 5,000-person company materializes on the edge once, not 5,000 times from origin) and object storage for the cold tail.

Why push-is-a-hint, pull-is-the-truth: notifications can be lost, duplicated, or arrive before a device reconnects — and none of it matters, because correctness lives entirely in the cursor pull. A device that misses every notification for a week converges identically on its next poll. The push path is allowed to be sloppy because the pull path is exact; that division is what keeps the notification fan-out cheap.

8. Deep dives

Conflicts: when two devices edit offline [R5]

The commit in §6 step 5 checks parent == current version. Laptop and desktop both edit report.docx (v17) offline; laptop reconnects first, commits v18. Desktop’s commit arrives claiming parent v17 — but current is now v18. Rejected.

What happens next is a product decision wearing an engineering costume, and the file layer has exactly one safe move: keep both. Desktop’s version commits as a sibling — report (conflicted copy, desktop, June 12).docx — and syncs everywhere like any file. Nobody’s work is destroyed; a human merges, because at the file layer the system has no idea what the bytes mean. Auto-merging is possible exactly when the system understands the format — which is the collaborative-editor question, a different design with a different data model (operations, not chunks).

The engineering content: conflict detection is the optimistic parent check (cheap, no locks, no coordination while offline — compare the distributed lock walkthrough’s machinery for when you can’t tolerate siblings); conflict resolution is explicitly punted to humans, and that punt is the correct call. Last-writer-wins here means a sales deck someone polished for four hours silently ceasing to exist.

Dedup, and its security asterisk [R1]

Content addressing makes dedup a side effect: step 2 of every upload asks “do you have hash H?” and identical chunks — across versions, users, the whole planet — store once with a reference count. At exabyte scale this is tens of percent of the storage bill.

The asterisk worth naming: cross-user dedup leaks one bit. If the server says “already have it” for a chunk you’ve never uploaded, you’ve learned someone has that exact content — the confirmation-of-file attack. And client-side end-to-end encryption (different keys → different ciphertexts) destroys cross-user dedup entirely. Real systems pick a point on that line: dedup within an account/enterprise only, or convergent encryption (key derived from content) which restores dedup but reintroduces the confirmation leak. Saying “dedup scope is a privacy decision, not a storage decision” is a Staff-level sentence.

Garbage collection: the hardest easy problem [R4]

Chunks are shared by versions, files, and users — so deletion is reference counting, and refcounting across a sharded metadata tier and an object store is where correctness goes to die. The safe shape: deletes only decrement; nothing is removed inline. A background GC walks candidates (refcount zero), re-verifies against the metadata source of truth, honors the 30-day version retention and a generous grace window (in-flight uploads in §6 may reference a chunk before any manifest commits — the ask-then-upload race), then deletes. GC bugs are the only failure in this system that destroys data while everything is green; run it conservative, audited, and reversible (tombstone before purge).

Sharing: mounts, not copies [R3]

A shared folder is a namespace mount: the folder subtree lives in its own namespace with its own sequence numbers; members’ roots contain a mount pointer plus an ACL row. Sharing copies nothing — members’ delta pulls simply span an extra namespace. ACL checks ride every metadata call (cheap, cached); content URLs are short-lived signed tokens minted per chunk fetch, so object storage never makes authorization decisions. A 5,000-member shared folder is then just a namespace with 5,000 cursors — and one commit notification fanning out to 5,000 devices, which is the news-feed problem at toy scale and solved the same way (notify online devices; offline ones catch up by cursor).

9. API design [R1, R2]

POST /v1/uploads:check                       [R1 · dedup + resume probe]
  body:    { namespace, path, chunk_hashes: [h1..hN], parent_version }
  returns: { missing: [h3], upload_session }

PUT  /v1/uploads/{session}/chunks/{hash}     [R1 · content, verified on receipt]

POST /v1/uploads/{session}:commit            [R1, R4]
  body:    { manifest: [h1..hN], parent_version }
  returns: { version } | 409 { conflict, current_version }

GET  /v1/delta?namespace=...&cursor=4711     [R2 · the sync primitive]
  returns: { changes: [...], new_cursor, has_more }

GET  /v1/chunks/{hash}?token=...             [R2 · signed, CDN-cacheable]

Two non-obvious decisions:

The 409 on commit returns the current version. A conflicting client immediately has what it needs to create the conflicted copy locally (§8) without a second round-trip — the conflict path is a first-class API outcome, not an error to retry.

/delta is the only sync read. There is no “list folder” sync path — folders, renames, deletes, shares all arrive as delta entries against the cursor. One primitive, one ordering, one catch-up story for every kind of change; clients that sleep a month and clients that never sleep run the same code.

The API hides chunking economics, dedup, CDN routing, and shard topology. It deliberately exposes content hashes — the client’s local store speaks the same addressing scheme, and that shared language is what makes delta sync a set difference instead of a protocol.

10. Data model and storage

Requirements → components

Access frequency

Storage technology choices

Access pattern matrix

Schemas

Metadata (per namespace shard) [R2–R5]

files:
  file_id, namespace_id, path, current_version, deleted_at?

versions:
  file_id, version, manifest: [chunk_hash, ...], size,
  author_device, parent_version, committed_at

change_log:
  namespace_id, seq (monotonic), change: {type, file_id, version}
  -- the delta API reads this; cursors index into it

mounts / acls:
  namespace_id, member_user, role, mounted_at_path

Chunk index [R1, R4]

chunks (KV):
  key:    sha256
  value:  { size, storage_tier, refcount, first_seen, gc_state }

11. Failure modes

Grouped by what they threaten. The pattern: name what fails, name what degrades, name what doesn’t.

Consistency failures

Client dies mid-upload

Fails: one upload session. Degrades: nothing — content-first ordering means the worst residue is orphan chunks, swept by GC after the grace window. The next attempt’s existence check resumes at the missing chunk. Doesn’t degrade: the namespace — no manifest committed, no version exists, no other device ever saw a half-file.

Concurrent offline edits

Fails: the linear version history of one file. Degrades: into two files — the conflicted copy (§8), visible, annoying, and lossless. Doesn’t degrade: either author’s bytes. The contrast to name: LWW here is silent destruction of human work, and “annoying but lossless” beats “clean but lossy” every time at the file layer.

GC races an in-flight upload

Fails: potentially, a chunk another upload is about to reference (checked present in §6 step 2, manifest not yet committed). Degrades: prevented by the grace window — chunks are uncollectable for hours after last reference or sighting; the commit re-validates manifest hashes and fails safe (client re-uploads the chunk) if GC won anyway. Doesn’t degrade: committed versions — GC never touches a chunk with a live manifest reference, verified against the source of truth at delete time, with tombstones before purge.

Availability failures

A metadata shard goes down

Fails: syncs, commits, and shares for the namespaces on that shard. Degrades: cleanly along the shard boundary — other users unaffected; affected clients keep serving local file access (the client holds a full replica of its folders — the offline path is the outage path) and queue commits. Doesn’t degrade: content durability or downloads of already-known chunks (signed URLs in hand keep working).

Notification service down

Fails: the wake-up path. Degrades: sync latency — from seconds to the clients’ fallback poll interval. Nothing else: pull-is-the-truth (§7) means correctness never depended on pushes. This is the failure mode that validates the hint/truth split — name it as such.

Scale failures

Hot shared file

Fails: nothing, if the CDN is doing its job — a 10K-member company all-hands recording is one origin fetch and 10K edge hits, because chunks are immutable and content-addressed (perfectly cacheable, integrity-checkable, key never reused). What to watch instead: the metadata side — 10K delta pulls and one namespace’s change_log getting hot; mitigated by read replicas for delta queries and notification batching.

12. What I’d skip, and say I’m skipping

Time check — five minutes left. Things I’d explicitly defer:

• Real-time co-editing. Different unit of change (operations vs chunks), different consistency machinery — the collaborative-editor walkthrough. The two systems coexist: this one stores the document; that one edits it.
• Client internals. Filesystem watchers, local chunk cache eviction, bandwidth throttling — a real engineering org lives there; none of it changes the server design.
• Search and previews. Pipelines consuming the change log (index text, render thumbnails) — downstream systems on an event feed this design already emits.
• End-to-end encryption. I’ve named the dedup tension (§8); designing the key hierarchy is a security interview.
• Compliance machinery. Residency pins namespaces to regions (the sharding model supports it by construction); legal hold suspends GC. Sentences, not sections.

Saying “I’d skip this, and here’s why” is a strong signal. It shows you know the full surface and are making deliberate scoping choices.

Interview flow summary

Architecture at a glance

UPLOAD (write)                          SYNC (read)

Client hashes chunks locally            Notification: "namespace changed"
 ↓                                       ↓
"Which hashes are new?" (dedup)         Pull delta since cursor
 ↓                                       ↓
Upload missing chunks → object store    Diff manifests vs local chunks
 ↓ (verify hash on receipt)              ↓
Commit manifest (parent check) ────►    Fetch missing chunks (CDN if hot)
 ↓ 409 conflict → conflicted copy        ↓
Notify other devices                    Apply locally, advance cursor

The walkthrough order for your whiteboard

1. Requirements       — sync not transfer; the two freshness/durability tiers
2. Capacity           — exabytes of content vs trillions of metadata rows
3. Evolution          — whole files → chunked manifests; when chunking earns it
4. Core choice        — content addressing; fixed vs CDC chunking
5. Upload path        — hash → check → upload → commit; ordering rationale
6. Sync path          — hint vs truth; cursors; CDN for hot chunks
7. Deep dives         — conflicts, dedup privacy, GC, mounts
8. API                — three-step upload, 409-with-payload, one delta primitive
9. Data model         — manifests, change log, chunk refcounts
10. Failure modes     — consistency / availability / scale groups

If the interviewer pushes deeper

13. Wrap-up

One crisp sentence before the interviewer’s next question:

This design splits every file into immutable content-addressed chunks and a transactional metadata trail, so dedup, resume, delta sync, and version history all fall out of one representation choice — and the only conflicts left are the human kind, which the system preserves instead of pretending to resolve.

What separates levels on this question

• SDE II uploads whole files to object storage with a files table, adds a polling client, and handles conflicts by overwrite — a working v1 that re-ships gigabytes for one-line edits.
• SDE III designs the chunk/manifest split with content addressing, gets the commit ordering right and can say why, builds sync as hint-plus-cursor, rejects LWW with the conflicted-copy argument, and prices dedup into the storage estimate.
• Staff/Principal treats GC as the highest-risk component and designs it conservative-by-construction; names the dedup/privacy/encryption triangle; shards metadata by namespace because of which transactions must be local; designs the change log as the product’s spine (and its compaction as the recovery story); and knows where this system ends and the collaborative editor begins.

The difference isn’t knowledge. It’s choosing which failures are allowed to exist — orphans over dangling pointers, duplicate files over merged losses, delayed pokes over trusted pushes. The strong answer is a chain of worse-failure-eliminated decisions.

Further reading

• System Design Interview Vol. 1 — Alex Xu. Chapter 15 (Design Google Drive) is the textbook treatment of this question.
• Streaming File Synchronization — Dropbox engineering. The block-level sync protocol from the team that shipped it.
• Inside the Magic Pocket — Dropbox’s exabyte-scale block storage: what Level 4 storage economics actually look like.
• The rsync algorithm — Tridgell & Mackerras. The rolling-hash delta lineage behind content-defined chunking.
• Designing Data-Intensive Applications — Kleppmann. Chapter 7 (transactions) for the metadata tier’s invariants; Chapter 12 for change-log-as-spine system shapes.
• Walkthrough: Designing a Collaborative Editor — what conflict handling becomes when the system does understand the bytes.