What happened
We tried pijul on a KeibiDrop mount. Git already works over the mount, so a second version control system seemed like a cheap win. It wasn't. With two peers recording into one shared repository, the second peer stopped seeing the first peer's changes. Cloning a repository through the mount works. Pulling an update recorded after the clone does not: the reader keeps seeing the old state.
We reproduced both failures in an integration test, then traced every notification between the peers. Pijul's change files arrive on the other side, because pijul happens to write those atomically: temp file, then rename into place. The pristine, rewritten in place, never announces itself at all. The writer's side sends adds and renames, and not one edit.
Why it happens
Pijul stores its state in a single B-tree file (Sanakirja) that it maps into memory and rewrites in place. That is the design of most embedded databases: LMDB, SQLite in mmap mode, and every store built on them.
mmap does not go through the filesystem's read and write path. It talks to the kernel page cache directly. On one machine that is exactly why it is fast: there is one cache and everybody shares it. Across two machines there are two page caches and no protocol between them. The writer's dirty pages flush whenever msync decides. The reader's clean pages have no reason to be dropped. Both kernels behave correctly, and the combined system is wrong.
On our mount this breaks in two places at once. The writer's dirty pages flush after the file is already released, outside the window where our change detection watches writes, so the peer is never told the file changed. And the reader tracks which byte ranges of a file it already holds; nothing arrives to invalidate them, so it serves the old pristine forever. Turning on kernel-level invalidation (macFUSE's auto_cache) changes nothing, and we tested both ways: the kernel cannot refresh a cache that our own layer above it still believes is valid.
Everyone who ships a network filesystem hits this. NFS's close-to-open consistency explicitly does not cover mmap across clients. SQLite's own documentation warns against putting databases on NFS. The LMDB docs say plainly: do not use LMDB on a network filesystem. This is not an implementation bug in any of them. It is the semantics of mmap meeting the reality of two kernels.
Git never hits it, and that is not luck. Git's object store is immutable: new data means new files, written once and renamed into place. Nothing is rewritten in place, so a stale page cache can only show you an old file list, never a torn database. That is why git clone and pull work over KeibiDrop today and pijul does not yet.
The known fixes, and their prices
- Turn off caching (direct_io). Every read goes to the source, always coherent. But mmap needs the page cache to exist, so mmap dies with SIGBUS. You fix the database by making it impossible to open.
- Invalidate on attribute change (macFUSE's auto_cache, NFS attribute caching). The reader drops cached pages when it notices the file changed. Helps, but it is polling with races, and it punishes local mmap writers.
- Leases and delegations (NFSv4, SMB oplocks). The server tells clients when their cache is invalid. This is the real machinery, and it is a protocol redesign, not a mount option.
- Restructure the data. Immutable files plus rename, or one replica per user synchronized by pulls. This sidesteps coherence instead of solving it. It is the git answer.
Where KeibiDrop stands
Our read path tracks which byte ranges of a file are already cached, and the peer announces every change. The pijul test pins down both halves of the failure: the writer's in-place mmap rewrite lands outside the window our change detection watches, so no edit notification is ever sent, and the reader's cached ranges are never invalidated. The fix is symmetric. Detect mmap writeback on the writer, and invalidate by chunk hash on the reader, dropping exactly the ranges that changed.
Until that lands: git works today because git never rewrites in place. Pijul trips the same stale-cache wall even when pulling between repositories, so treat a pijul repository on the mount as read-only for now, and do not point two writers at one mmap'd database through any network filesystem, ours included.