GPB Sync (macOS)

A menu-bar backup client for a self-hosted GPB server. It scans Apple Photos, watched folders and dropped files, hashes them, declares them to the server, and then does what the server tells it — sending the bytes over the LAN straight to an Android node when there is a verified link, and through the server when there is not.

The specs live outside this repo, with the other two repos' copies. Bare section references like §5.6 and §7.6 point at the v3 relay contract (DESKTOP_SYNC_APP_GUIDE.md), which is not superseded — the relay path is a permanent, first-class fallback. References like client plan §5.1 and architecture §7.2 point at V4_CLIENT_PLAN.md and V4_ARCHITECTURE.md; where those two disagree, the architecture wins. Earlier drafts of all of them are in this repo's git history rather than its working tree.

What v4 changed

v3's client was an uploader that decided things: it scanned, hashed, asked which items to skip, and pushed the rest. The server learned about a file when its bytes arrived.

v4's client is a source that follows orders:

1. it scans, hashes and declares items — metadata only, zero bytes;
2. it long-polls a control channel and executes the commands it gets;
3. it pushes bytes to whichever destination it is told — a node over mutual TLS, or the server's chunked relay;
4. it reports what happened, honestly, including the byte counts.

The server decides what transfers, when, over which route, and to where. The client never moves an item to done on its own: transferred is as far as it goes, and the item is closed out when a later declare answers ALREADY_KNOWN.

It does not take custody of anything. An earlier revision had the client hold every declared original — copying photo-library originals to disk, budgeting the copies at 25 % of free space, evicting the largest and re-exporting on demand — on the reasoning that a declared item's only copy is local. The reasoning was sound and the conclusion was not: that is a large, failure-prone mechanism whose best outcome is preserving files the user already has, and whose worst was a default budget of zero that failed every Photos item on a fresh install. The owner is a better custodian of their own photos than a quota is. So transfers now open an ItemStream, read chunk-sized windows from it, and close it — folder items stream from their real path with no copy at all, and a Photos original becomes a scratch file that is deleted the moment that transfer ends, success or failure. No user file is ever moved or deleted.

A v3 server still works. The client tries /api/v4/client/sync, and a 404 hands any parked items back to the v3 queue and runs exactly as it did before. That downgrade is a park, not a switch: v4 is retried 30 minutes later, the same interval the Android node uses. A staged rollout, a backend restart or a proxy answering for the origin must not cost the LAN path until someone relaunches the app — degrading with no flag day has to work in both directions.

Only the origin's own JSON counts as evidence about what the server speaks. Every reply the server produces for an /api/ path is JSON, its 404 included, so a non-JSON body — whatever status it carries — came from something between client and server and is retried as a transient fault. That includes a non-JSON 404: Traefik answers 404 page not found for the seconds a container is being recreated, so reading it as "no v4 here" parked both this client and the Android node for half an hour after every redeploy, which meant every attempt to test a fix watched a client that had stopped speaking v4 before the fix landed.

Build

xcodebuild needs an explicit toolchain here because xcode-select points at the Command Line Tools:

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcodebuild -project gpb-sync.xcodeproj -scheme gpb-sync -configuration Release -destination 'platform=macOS' build

The target uses a file-system-synchronized group, so any .swift file added under gpb-sync/ is compiled automatically — no project edits needed.

Test

Tests/run-tests.sh

Three stages, ~1 minute, nothing touches a real server:

1. Forbidden-API guard — fails if URLSession.shared reappears (its 60 s request timeout is what killed every large upload) or if a media file is read whole into memory.
2. Unit checks (988) — chunk arithmetic, dedup keys verified against the openssl recipe from the guide, the chunk bitmap, the full error taxonomy, backoff/circuit breaker, and the SQLite journal's state machine. The v4 half covers endpoint parsing, resume from a bounded missing list, the LAN error codes, DER/PKCS#10 encoding against the byte sequences in RFC 5480, certificate renewal timing, the task-update outbox's collapse-and-ack rules, the declare → assign → transfer → release path, and a check that no raw MAC or SSID can reach the wire form.
3. Integration checks (29) — the real APIClient and Uploader driven against Tests/mock_server.py, which enforces the backend's actual rules (chunks are PUT-only, zero-byte parts are never counted as received, /finish reassembles and re-hashes). Covers single-shot, chunked, resume, a failing chunk, cannot finish: N missing, and the /finish recovery protocol.

The guard is not an Xcode build phase: the project has ENABLE_USER_SCRIPT_SANDBOXING = YES, which denies a script phase read access to the source tree. Run it from Tests/run-tests.sh or CI instead.

Layout

gpb-sync/
  App/         AppController        @MainActor view model: triggers, pairing, UI state
  Core/        AppConfig, CredentialStore, Hashing, AppSupport
  Net/         Net (the one URLSession), APIClient (the one send()), Failure,
               APIModels, V4Models, V4APIClient
  Store/       Store (actor over SQLite), V4Store, SQLiteDB, ItemRecord
  Engine/      SyncEngine (actor), SyncEngineV4, Uploader, UploadLimiter,
               Backoff, SyncSnapshot
  Control/     ControlChannel       the heartbeat loop and command dispatch
  Transport/   Destination + RelayDestination + PeerDestination, TransferDriver
  Security/    SecureStore, ClientIdentity (keys, CSR, certs), PeerTrust, DER
  Rendezvous/  NetworkInspector, ProbeRunner, NodeBrowser, Peer
  Logging/     Log (OSLog + rolling JSONL + events table), Diagnostics
  Sources/     PhotosSource, FolderSource, FSEventsWatcher, ItemStream
  Views/       MainWindowView, StatusTab, PeersTab, FailuresSheet, SkippedSheet,
               MenuBarView, PairingView

Portability

The client plan asks for a GPBKit package so a Windows or iOS target is a matter of implementing protocols rather than forking logic. What is here is the seam without the package split: SecureStore and NetworkInspector are protocols with macOS implementations (KeychainSecureStore, DarwinNetworkInspector), and everything in Engine/, Transport/, Store/ and Control/ is written against them plus Foundation. Extracting a real package later is then a file move rather than a redesign. The seams that actually matter were the point; moving 30 files between targets to prove it was not worth the merge conflicts today.

The iOS constraints are designed around already, not deferred: transfer is push-only, the fingerprint tolerates having no gateway MAC and no SSID, and NetworkInspector returning nothing but a subnet is a supported answer rather than a failure.

The load-bearing rules

• One URLSession. Net.session, with a 120 s inactivity timeout, a 24 h resource ceiling, and per-endpoint URLRequest.timeoutInterval overrides. /upload/finish gets max(600, size_GB × 180) seconds because the server re-reads and SHA-1s the whole file before it answers, with no bytes on the wire.
• One send(). Every request goes through APIClient.send: credential injection, a fresh X-Request-ID (echoed back and stored in the server's own logs), a JSON content-type assertion, error classification, and exactly one structured log line. It never retries internally — retries belong to the scheduler, where the backoff is visible and bounded.
• totalChunks = ceil(size / chunkSize), never + 1. A trailing zero-byte part is stored by the server but never counted as received, which produces cannot finish: 1 of N missing forever. Every PUT is self-checked: the index just sent must appear in the response's received array.
• A timeout is an unknown outcome, not a failure. After the bytes are sent, the recovery protocol asks /status, then dedup/check, and only re-uploads when the server demonstrably does not have the content.
• The journal is SQLite. uploadId and a chunk bitmap are written after every acknowledged part, so a crash resumes mid-file at zero cost. Every counter in the UI is a SQL aggregate, so the tiles cannot drift.
• Nothing that touches bytes is @MainActor. The engine is an actor; workers pull from the persistent queue through a live-resizable limiter, with separate lanes for small files (default 4) and large ones (2), 3 parallel chunks inside a large file, and at most 2 Photos exports at a time.
• One chunked uploader, two destinations. TransferDriver owns the chunk arithmetic, the bitmap, resume, concurrency, progress, cancellation and the finish-recovery protocol, written once against the Destination protocol. RelayDestination speaks the v3 API; PeerDestination speaks the node's receiver. Two uploaders would mean carrying every fix twice — and §5.6 records what one copy of the chunk arithmetic getting it wrong already cost.
• Both, not either. A node's certificate must chain to the owner CA and match the exact fingerprint the server named. There is no plaintext fallback and no "retry without validation": a rejection is reported as PEER_UNTRUSTED, and it is the one failure the user is warned about.
• The client never declares its own success. transferred is as far as it goes. Thirty minutes later — the server's own stall-watchdog interval — the item is declared again: ALREADY_KNOWN closes it out for good, and a NEEDS_TRANSFER means the transfer really was lost and is worth re-planning. That round trip replaced RELEASE_ITEM, and it is also the only "don't send this again" mechanism there is.
• A canary proves the pipe before a real photo uses it. A VERIFIED link proves one 200-byte GET completed, which is not the same fact as "a transfer works" — in production the difference was sixteen consecutive failures against a node the server called healthy. So the server sends PUSH_LOCAL{purpose:CANARY} with 64 KiB of synthetic bytes inline, and the client runs them down the ordinary path: same destination, same mTLS, same grant, same init/chunk/finish. The client decides nothing about the result — it does not gate its own queue, retry on its own schedule, or remember the outcome beyond a line in the Peers tab. The server owns the retry budget.
• Reaching a peer gets a longer retry ladder than reaching the server. Five attempts at 2/4/8/16/32 s for connection establishment, against four at 2/5/12/30 s for an ordinary request: a Mac waking from sleep, a node roaming between bands and a receiver rebinding on a new port are all measured in tens of seconds, and the request ladder gives up inside the first of them.
• bytesSent is the true total on every COMPLETED. It used to default to zero and every call site took the default, so 61 of 68 completed LAN transfers reported nothing and the server's WAN-savings counter — the number that says what the whole LAN path is worth — was computed from a column of zeroes. The parameter has no default now, so a call site cannot forget.
• Nothing is dropped silently. A file rejected here for not looking like media, or by the server for being empty or unsupported, lands in the Skipped sheet with its reason. A silently skipped photo is indistinguishable from a lost one.
• The Wi-Fi name needs Location access, and is worth asking for. macOS counts an SSID as a location, so CWInterface.ssid() returns nil without authorisation — which this app never requested, so the field was silently always empty. It matters because the fingerprint is scored (§4.2) and Android cannot read the ARP table: the node reports no gateway MAC, this Mac reported no SSID, and a genuinely co-located pair scored exactly the probe threshold of 10, clearing it only because the comparison is <. A guest VLAN or two APs on different subnets took it to 0 and the pair then never probed. Publishing the SSID hash puts a signal on both sides and takes the match to 35. Asked for on the first heartbeat that builds a fingerprint, not at launch; a refusal costs one signal and nothing else. No position is ever requested.
• Local network access is asked for at launch. macOS gates the LAN behind a privacy switch with no API to read or request it — the system prompts the first time the app touches the local network, and only then. The only thing here that does is the mDNS browse, which lives inside the control channel and so needs a paired client first; left alone, the dialog arrives days later, mid-transfer, on an unattended Mac, and a missed prompt costs the LAN route silently and for good. LocalNetworkAccess.prime() starts a throwaway Bonjour browse at launch instead, which is the request. Only an explicit refusal (kDNSServiceErr_PolicyDenied, or EPERM) is reported as denied — a network that merely filters multicast fails identically, and sending that user to a settings pane would be wrong. NSLocalNetworkUsageDescription and NSBonjourServices must both be in the built bundle; without the latter macOS refuses the browse outright rather than prompting.
• close() on every exit path, cancellation included. With holds gone, the one hazard the budget was genuinely guarding against is a leaked scratch file: a 4 GB export per failure is how a disk fills. Every stream is opened in a defer-closed scope, and closing twice is safe. The old Application Support/GPB Sync/Holds directory is deleted once, on the first launch after upgrade — it is the app's own scratch, which is the only reason that is safe to do.

Deliberate deviations from the guide

• Credentials live in a 0600 file in the sandbox container, not the Keychain: a locally-signed build gets a new code signature on every rebuild, which makes the Keychain prompt for the login password at each launch. CredentialStore is the only thing to change if the app is later signed with a stable team.
• The local-transfer private key has to stay in the Keychain — mTLS needs a SecIdentity, which needs a keychain — so it meets the same problem head-on. The data protection keychain would avoid it entirely, but wants an application-identifier entitlement only a team-signed build has; an ad-hoc build gets errSecMissingEntitlement (-34018), and keychain-access-groups is not a way in either — an ad-hoc binary claiming one is killed at launch. So the key lands in the login keychain, whose ACL names the signature that created it, and the next build is a different signature: hence the password prompt for a key macOS can only call <key>. Two rules answer it. Keychain interaction is off for the whole process, so a mismatched ACL fails cleanly instead of blocking the app behind a modal. And a key is tested by signing with it, not by finding it: one that refuses is purged, regenerated, and the leaf issued against it dropped, which puts a CSR on the next heartbeat and has local transfer back unattended. Signing with a stable team makes all of this dead code, and the data protection keychain is already the first path tried.
• items(dedup_key, target_account) is a plain index, not the unique one in the guide's schema: two different items can legitimately hold identical bytes (the same photo in Photos and in a watched folder), and §5.1 tells clients to expect duplicate keys in one batch. A unique index would refuse the second row.
• A task that never finds a network path is cancelled after 120 s. Combining waitsForConnectivity = true with a 24 h resource timeout — both prescribed — otherwise allows a request to stall silently for a day.
• Every request carries an app-side watchdog (2 × timeoutInterval + 30 s), because URLRequest.timeoutInterval cannot be relied on. Observed in production: one chunk PUT of a 226-part upload sat in flight for 13 minutes against a 180 s timeout that never fired, while its neighbours on the same connection finished in 1.5 s each — an HTTP/2 stream can stall while the connection still looks alive, and no delegate callback is made. The whole file waits forever behind that one part. The watchdog converts it into a normal timeout, and since chunk writes are idempotent (§5.4) the part is simply re-sent. GPB_WATCHDOG_SCALE shrinks the deadline for tests.
• A failed Photos export is only permanent when the asset is gone. An iCloud original that did not download, or a full disk, is transient — classifying those as permanentItem dead-lettered ten large videos and tripped the circuit breaker on what was a temporary iCloud problem.
• The mTLS private key does live in the Keychain, unlike the pairing credentials. It has to: URLSession needs a SecIdentity for a client certificate, and that means a keychain-resident key. KeychainSecureStore tries the data protection keychain first and falls back to the file keychain, and if both refuse it says so and the app runs relay-only. Losing the LAN route is a performance regression; guessing at key storage would be a correctness one.
• RECEIVER_PAUSED is peerCapacity, not capacity. The client plan's table maps it to "capacity", but this codebase's capacity class pauses the entire run — right for a full server, wrong for one phone coming off its charger. peerCapacity is item-scoped: the partial is kept, no attempt is burned, and the server re-routes or waits.
• The peer transport gets its own URLSession per node, which is the one deliberate exception to "one URLSession". Pinning and client certificates are delegate-driven and Net.session has no delegate — nor could it have a per-node one. PeerSessionPool keeps exactly one per (node, pinned fingerprint) and tears it down when either changes.
• Info.plist is a partial file at the project root. NSBonjourServices is array-valued and has no INFOPLIST_KEY_ mapping, and without it macOS refuses the mDNS browse outright. GENERATE_INFOPLIST_FILE stays on and merges into it. It lives outside gpb-sync/ because that directory is a file-system synchronized group and a plist inside it would also be copied in as a stray resource.
• A truncated missing-list is refused, not believed. The node answers a status query with a count plus at most 64 missing indices, so on a 1280-chunk file that list is a sample. Reading it as an inventory would mark un-sent chunks present and assemble a file with holes; when the list does not account for every gap the count implies, the client re-sends instead. Slower, and the only safe reading.