The failure
Correction, July 30. The diagnosis below was wrong in one important place, and instrumenting every layer found the real chain. The writer announces the rewritten pristine just fine. What broke was on the reader: our own truncate-on-open stamped the remote file's metadata with the local clock, so every genuine update looked stale and was rejected; a lock-taking read-write open also announced the file back, silently taking authorship from its owner; and beneath both, the kernel page cache kept serving old pages. Three stacked caches, each hiding the next. Both bugs are fixed, and pulling between pijul repositories over the mount now works, with live collaboration mode on. The general lesson of this post stands, but the specific failure was self-inflicted. Details in the repository history.
We tried pijul on a KeibiDrop mount. Git already works over the mount, so a second version control system seemed like a cheap win. It wasn't. With two peers recording into one shared repository, the second peer stopped seeing the first peer's changes. Cloning a repository through the mount works. Pulling an update recorded after the clone does not: the reader keeps seeing the old state.
We reproduced both failures in an integration test, then traced every notification between the peers. Pijul's change files arrive on the other side, because pijul happens to write those atomically: temp file, then rename into place. The pristine, rewritten in place, seemed never to announce itself. That reading turned out wrong; see the correction above. What is true: the reader kept serving the old pristine no matter how long we waited.
The cause
Pijul stores its state in a single B-tree file (Sanakirja) that it maps into memory and rewrites in place. That is the design of most embedded databases: LMDB, SQLite in mmap mode, and every store built on them.
mmap does not go through the filesystem's read and write path. It talks to the kernel page cache directly. On one machine that is exactly why it is fast: there is one cache and everybody shares it. Across two machines there are two page caches and no protocol between them. The writer's dirty pages flush whenever msync decides. The reader's clean pages have no reason to be dropped. Both kernels behave correctly, and the combined system is wrong.
On our mount the break was layered, and the correction above gives the corrected order: the reader's own metadata got stamped with local time, so update announcements were rejected as stale; the reader's byte-range cache therefore never invalidated; and the kernel page cache under both kept old pages alive. Kernel-level invalidation (macFUSE's auto_cache) is necessary but could do nothing while the layer above it rejected the updates.
Everyone who ships a network filesystem hits this. NFS's close-to-open consistency explicitly does not cover mmap across clients. SQLite's own documentation warns against putting databases on NFS. The LMDB docs say plainly: do not use LMDB on a network filesystem. This is not an implementation bug in any of them. It is the semantics of mmap meeting the reality of two kernels.
Git never hits it, and that is not luck. Git's object store is immutable: new data means new files, written once and renamed into place. Nothing is rewritten in place, so a stale page cache can only show you an old file list, never a torn database. That is why git clone and pull worked over KeibiDrop from the start, while pijul needed the fixes above.
The known fixes, and their prices
- Turn off caching (direct_io). Every read goes to the source, always coherent. But mmap needs the page cache to exist, so mmap dies with SIGBUS. You fix the database by making it impossible to open.
- Invalidate on attribute change (macFUSE's auto_cache, NFS attribute caching). The reader drops cached pages when it notices the file changed. Helps, but it is polling with races, and it punishes local mmap writers.
- Leases and delegations (NFSv4, SMB oplocks). The server tells clients when their cache is invalid. This is the real machinery, and it is a protocol redesign, not a mount option.
- Restructure the data. Immutable files plus rename, or one replica per user synchronized by pulls. This sidesteps coherence instead of solving it. It is the git answer.
Our position
Our read path tracks which byte ranges of a file are already cached, and the peer announces every change. The pijul test pinned down the failure to acceptance, not announcement: the reader's staleness check compared against metadata its own operations had overwritten with local time. The fix is a watermark: remote updates are judged only against the peer's last announced time, which no local operation can touch, plus a gate so that opening a file for writing without writing to it never announces anything.
Where it stands now: git works, and pulling between pijul repositories over the mount works too, with live collaboration mode enabled. Two writers on one mmap'd database through any network filesystem, ours included, remains unsupported: the database's own lock files replicate asynchronously, so no cross-machine exclusion exists to rely on.