Cross-Platform File Sync: The Hidden Complexity

macOS atomic saves, Windows file locking, and Linux FUSE variants

9 min read | KEIBIDROP Series

The Problem

KEIBIDROP presents a virtual folder on your desktop; you drop a file in, and it shows up on the other machine, encrypted and peer-to-peer with no cloud involved. The networking and encryption were the straightforward part, but the filesystem is where the real complexity lives.

Every operating system has a different opinion about how files should behave. And every application on those operating systems has its own interpretation of the filesystem API.

macOS: Where Simplicity Hides Complexity

macOS uses macFUSE for userspace filesystems. It works, but macOS has several behaviors that make file sync difficult. Development started on an Intel Mac, where these issues showed up early.

Extended Attributes Everywhere

macOS stores metadata in extended attributes (xattrs) for almost everything: Finder tags, quarantine flags, download origin, Spotlight indexing data. Every time you interact with a file in Finder, it may read or write several xattrs. Your FUSE filesystem must handle getxattr, setxattr, and listxattr correctly, or Finder will behave unpredictably.

One specific xattr required special treatment: com.apple.quarantine. Sandboxed apps like Preview.app check this attribute before opening files. On a FUSE mount, Gatekeeper cannot complete its verification and silently refuses to open the file. The fix is to block that xattr entirely; see the implementation in Getxattr() and Listxattr(). For the full story, see macOS Preview and FUSE.

The TextEdit Save Flow

This took a full day to debug. When you save a file in TextEdit (and many other macOS apps), here is what actually happens:

  1. Write new content to a temporary file: .myfile.txt.sb-12345
  2. Call renamex_np() with RENAME_SWAP to atomically swap the temp file and the original
  3. Delete the old temporary (which now contains the previous version)

This atomic swap ensures that no reader ever sees a partially written file. For a sync tool, it means that a simple "file changed" event is actually a rename, a swap, and a delete. You need to figure out that the net result is "the file content was updated." The Rename handler deals with this. 

// What the sync layer sees:
// 1. Create:  .myfile.txt.sb-12345
// 2. Write:   .myfile.txt.sb-12345  (new content)
// 3. Rename:  myfile.txt -> .myfile.txt.sb-67890  (atomic swap)
//             .myfile.txt.sb-12345 -> myfile.txt
// 4. Delete:  .myfile.txt.sb-67890
//
// What actually happened: the user pressed Cmd+S

Finder Expectations

Finder calls getattr constantly. It expects sub-millisecond responses. If your getattr implementation needs to check with a remote peer, Finder will show the spinning beach ball. We cache all metadata locally and update it asynchronously.

negative_vncache: macOS caches ENOENT (file not found) results aggressively. If Finder checks for a file before it exists and gets ENOENT, it will cache that result and refuse to see the file even after it appears. You must be careful about the order of operations when making remote files visible. This is documented in the macFUSE mount options.

Windows: Different Rules Entirely

Windows uses WinFSP (Windows File System Proxy) as its FUSE equivalent. WinFSP is well documented and more consistent than macFUSE. However, Windows has fundamentally different filesystem semantics. We have not fully tested KEIBIDROP on Windows yet, but these are the known issues we are preparing for.

Mandatory File Locking

On Unix, file locks are advisory. A program can open a file even if another program has "locked" it. On Windows, file locks are mandatory and enforced by the kernel. If Word has a document open, no other process can write to it. Your sync daemon cannot update the file until Word releases the lock. 

// On macOS/Linux:
// Process A: open("file.txt", O_RDWR) -> success
// Process B: open("file.txt", O_RDWR) -> success (advisory lock ignored)

// On Windows:
// Process A: CreateFile("file.txt", GENERIC_WRITE, 0, ...) -> success
// Process B: CreateFile("file.txt", GENERIC_WRITE, 0, ...) -> ERROR_SHARING_VIOLATION

Path Semantics

Windows paths are case-insensitive but case-preserving. File.txt and file.txt are the same file. On macOS with APFS, the default is also case-insensitive, but Linux is case-sensitive. A file named README.md and another named readme.md can coexist on Linux but will collide on Windows and macOS.

MAX_PATH

The classic Windows limitation: paths cannot exceed 260 characters by default. Modern Windows supports long paths via a registry setting (LongPathsEnabled), but many applications still break with long paths. A sync tool must either enforce path length limits or handle the errors when applications fail.

POSIX Emulation Performance

If you use the Git Bash terminal emulator (MSYS2/MinGW POSIX layer) on Windows, file operations like cp are roughly 10x slower than the same operations via Command Prompt or PowerShell. This is because the POSIX emulation layer translates every syscall through an abstraction. For file transfer benchmarks and real-world usage, always test with native Windows tools.

Antivirus Interference

Windows Defender (and other antivirus software) scans every file that is created or modified. When your FUSE filesystem creates a new file, the antivirus will immediately open it for scanning. This triggers additional Open, Read, and Close operations that your filesystem must handle gracefully. File creation is noticeably slower as a result.

Linux: Freedom and Fragmentation

Linux has native FUSE support in the kernel. No third-party driver needed. Andrei handles the Linux testing, and the issues are different from macOS.

FUSE 2 vs FUSE 3

FUSE 2 and FUSE 3 have different APIs. Some distros ship FUSE 2, some ship FUSE 3, some ship both. The cgofuse Go library abstracts this to some degree, but there are behavioral differences. FUSE 3 supports readdirplus for better performance. FUSE 2 does not. If you use FUSE 3 features, you lose compatibility with older distros.

One concrete issue: Linux FUSE read-ahead fires parallel reads on a single gRPC stream. On macOS, reads are serialized. This caused data corruption on Linux until we added a mutex to serialize reads on the stream. The trade-off is that parallelism is lost; multiplexed streams are the proper fix, planned for a future release.

Library Paths

So far we have only tested on Debian/Ubuntu. The FUSE library is at /usr/lib/x86_64-linux-gnu/libfuse3.so there, but Fedora, Arch, NixOS, and Alpine each put it somewhere different. BSD, Solaris, and other Unix-likes have their own FUSE implementations with their own quirks. We have not touched any of those yet.

File Handles and Reference Counting

Multiple processes can have the same file open simultaneously. The filesystem must track how many open handles exist for each file, because you cannot safely delete or replace a file while it is open.

KEIBIDROP tracks this via OpenFileCounter and the OpenFileHandlers map: 

// Simplified from pkg/filesystem/types.go

type OpenFileCounter struct {
    count atomic.Int32
}

func (o *OpenFileCounter) Open() int32 {
    return o.count.Add(1)
}

func (o *OpenFileCounter) Release() int32 {
    return o.count.Add(-1)
}

The reference counting pattern is straightforward, but it interacts with every other part of the system. Rename while open? Delete while open? Sync update arrives while the file is being written? Each combination is a potential bug. See Write/Release Race Conditions for the details.

The Notification Protocol

When a file changes on one peer, the other peer must know about it. KEIBIDROP uses a gRPC Notify handler that processes file events: 

// From keibidrop.proto
enum NotifyType {
    DISCONNECT   = 0;
    ADD_DIR      = 1;
    REMOVE_DIR   = 2;
    ADD_FILE     = 3;
    EDIT_FILE    = 4;
    REMOVE_FILE  = 5;
    RENAME_FILE  = 7;
    RENAME_DIR   = 8;
}

FUSE operations (create, write, rename, delete) trigger outbound notifications via OnLocalChange. The receiving peer processes these in the Notify handler and updates its local filesystem accordingly. Renames are atomic (not decomposed into delete + add); see the technical deep dive for why.

The Non-FUSE Fallback

FUSE is not available everywhere. iOS and Android have no FUSE support. Some Linux servers run without FUSE installed. Corporate machines may have policies preventing FUSE mounts.

KEIBIDROP's non-FUSE mode exposes a programmatic API via pkg/logic/common/logic.go: 

// Add a local file to share with the peer
err := peer.AddFile("/path/to/document.pdf")

// List files available from the peer
files, err := peer.ListFiles()

// Pull a specific file from the peer
data, err := peer.PullFile("document.pdf")

The non-FUSE mode uses the same encryption, the same networking, and the same notification protocol. The only difference is that file operations are explicit API calls instead of implicit filesystem operations. This is what the agent CLI (kd) uses under the hood.

File state tracking in non-FUSE mode uses SyncTracker, which maintains LocalFiles and RemoteFiles maps with sync status flags.

Time Synchronization

File timestamps seem simple until you realize that two machines rarely agree on what time it is.

Instead of relying on wall clock time, we use a monotonically increasing version counter for each file. When a file is modified, its version increments. The peer with the higher version number has the newer content. This sidesteps clock synchronization issues entirely.

What We Would Do Differently

9 min read | KEIBIDROP Series | Technical Deep Dive | FUSE Deadlocks | Write/Release Race | macOS Preview + FUSE | Agent CLI