Why macOS Preview Can't Read Your FUSE Files

The Symptoms

When you double-click a PDF on a FUSE mount, Preview opens it, spins for a moment, and shows the dialog: "The file may be damaged or may not be a file format that Preview recognizes."

But the file is fine. You can verify it:

$ cat /mnt/keibidrop/report.pdf | wc -c
2847361

$ md5 /mnt/keibidrop/report.pdf
MD5 (/mnt/keibidrop/report.pdf) = a1b2c3d4e5f6...

$ file /mnt/keibidrop/report.pdf
/mnt/keibidrop/report.pdf: PDF document, version 1.7

$ open -a "Google Chrome" /mnt/keibidrop/report.pdf
# Works perfectly

cat reads every byte correctly. md5 checksums match. file identifies it as a valid PDF. Chrome opens it without issue. Only Preview fails. The same thing happens with images: .png, .jpg, .tiff; Preview rejects them all, while every other application reads them fine.

The Investigation

The first step was to trace Preview's system calls with dtruss to see what it was actually doing with the file. (On recent macOS, you need to copy Preview.app to /tmp first because SIP prevents tracing apps in /Applications.)

$ cp -R /Applications/Preview.app /tmp/Preview.app
$ sudo dtruss /tmp/Preview.app/Contents/MacOS/Preview report.pdf 2>&1 | grep -E "open|read|xattr"

The trace revealed something striking: Preview never calls read() on the file. It opens the file, queries extended attributes, then immediately closes it. The FUSE logs confirmed the same pattern:

Open      /report.pdf  flags=O_RDONLY   -> handle 7
Getxattr  /report.pdf  name=com.apple.quarantine
Flush     /report.pdf  handle=7
Release   /report.pdf  handle=7
// No Read calls. Not a single one.

Preview opens the file, checks for the quarantine extended attribute, and gives up. It never attempts to read the actual content. This is not a bug in our filesystem; Preview is deliberately refusing to read the file based on metadata checks that happen before any content access.

Issue 1: Sandboxed Apps Cannot Access FUSE Mounts

Preview.app is sandboxed. Since macOS Mojave, most system applications run inside the App Sandbox, which restricts filesystem access to specific locations. FUSE mounts, by default, are only accessible to the user who mounted them. The sandbox adds a second layer: even if the user has access, sandboxed apps are restricted to "approved" filesystem types.

By default, a FUSE mount is only accessible to the mounting user's UID. Sandboxed apps run with the same UID but go through an additional access check that evaluates the mount type. FUSE mounts with default permissions often fail this check silently; the app receives EACCES on the open() call but handles it by showing a generic error dialog rather than a permission error.

The fix: mount with allow_other and let the kernel handle permission checks:

// Mount options for macOS compatibility with sandboxed apps
opts := []string{
    "-o", "allow_other",
    "-o", "default_permissions",
}
fs.host.Mount(mountpoint, opts)

Source: OpenEx mount options

On macOS with macFUSE, allow_other is supported without additional configuration (unlike Linux, which requires editing /etc/fuse.conf). The default_permissions option tells the kernel to enforce standard Unix permission checks, so you are not giving up security by using allow_other.

Issue 2: Gatekeeper Quarantine

After fixing sandbox access, Preview's behavior changed slightly. It now successfully opens the file, but still shows the "damaged" dialog. The dtruss trace showed a new pattern: after open(), Preview calls getxattr() looking for com.apple.quarantine.

The com.apple.quarantine extended attribute is Gatekeeper's way of tracking files downloaded from the internet. When you download a file via Safari or Chrome, the browser sets this attribute. Gatekeeper-aware applications check for it and may refuse to open "quarantined" files that have not been explicitly approved by the user.

For FUSE files, there is no quarantine attribute, because the file was not downloaded from the internet. But when a FUSE filesystem returns an error code for getxattr that Preview does not expect, Preview interprets it as "this file is suspicious" and refuses to proceed.

The fix: return ENODATA (attribute does not exist) for the quarantine attribute, and handle Listxattr to not advertise attributes we do not support:

func (fs *KeibiFS) Getxattr(path string, name string, buf []byte) int {
    // Gatekeeper quarantine check: explicitly say "no quarantine"
    if name == "com.apple.quarantine" {
        return -fuse.ENODATA
    }

    // For other xattrs, try to read from the backing file
    localPath := fs.localPath(path)
    size, err := syscall.Getxattr(localPath, name, buf)
    if err != nil {
        return -fuse.ENODATA
    }
    return size
}

func (fs *KeibiFS) Listxattr(path string, fill func(name string) bool) int {
    // Do not advertise quarantine in the attribute list
    localPath := fs.localPath(path)
    attrs, err := listxattr(localPath)
    if err != nil {
        return 0
    }

    for _, attr := range attrs {
        if attr == "com.apple.quarantine" {
            continue // hide quarantine from listings
        }
        if !fill(attr) {
            break
        }
    }
    return 0
}

Source: Getattr handler

The key insight is that ENODATA means "the attribute does not exist," which is the correct response for a non-quarantined file. Other error codes like ENOTSUP or EIO cause Preview to assume the worst.

Issue 3: Preview Needs mmap, direct_io Breaks It

With sandbox access and quarantine fixed, Preview finally attempted to read the file content. But it still failed. The FUSE logs now showed a different pattern: Preview opened the file with O_RDWR (not O_RDONLY), and then the Read call returned data but Preview did not use it.

Preview opens files with read-write access because it supports annotations. Even if you never annotate a PDF, Preview opens it with write capability. Preview also uses mmap to access file content, the same mechanism git uses for pack files.

Our FUSE filesystem used direct_io for all files opened with write access (to ensure remote changes are always visible). But direct_io disables the page cache, which breaks mmap. Preview's mmap call would succeed (lazy allocation), but the first page access would trigger SIGBUS, which Preview catches internally and translates to "file may be damaged."

The fix: exempt document and image formats from direct_io, since these are the files Preview (and other macOS apps like Quick Look) open with mmap:

// File extensions that macOS apps commonly open with mmap.
// These must NOT use direct_io, even when opened with write access.
var mmapSafeExtensions = map[string]bool{
    ".pdf":  true,
    ".png":  true,
    ".jpg":  true,
    ".jpeg": true,
    ".tiff": true,
    ".tif":  true,
    ".gif":  true,
    ".bmp":  true,
    ".heic": true,
    ".webp": true,
    ".svg":  true,
}

func shouldUseDirectIo(path string, flags int) bool {
    // .git internals need mmap
    if strings.HasPrefix(path, "/.git/") {
        return false
    }

    // Document/image formats need mmap for Preview/QuickLook
    ext := strings.ToLower(filepath.Ext(path))
    if mmapSafeExtensions[ext] {
        return false
    }

    // Everything else uses direct_io for freshness
    if flags&(syscall.O_WRONLY|syscall.O_RDWR) != 0 {
        return true
    }

    return true
}

Source: shouldUseDirectIo

The Three-Layer Onion

Each fix alone does nothing, because the symptom is identical at every stage (Preview shows "file may be damaged") while the underlying cause is different each time.

If you fix only layer 1, Preview still fails at layer 2; if you fix layers 1 and 2, Preview still fails at layer 3. All three fixes must be applied simultaneously for Preview to work. This is likely why so many FUSE filesystem projects have open issues about Preview compatibility: a developer fixes one layer, sees no improvement, assumes the approach is wrong, and moves on.

Debugging Tips

If you are building a FUSE filesystem on macOS and applications are not reading your files, here is a systematic debugging checklist:

1. Check your FUSE logs for Read calls. If the app opens the file but never issues a Read, the problem is before the content layer. Look at Getxattr calls and access permission errors.
2. Look for quarantine queries. If you see Getxattr for com.apple.quarantine, make sure you return -ENODATA, not -ENOTSUP or -EIO.
3. Check your direct_io settings. If the app issues Read calls and gets data back but still fails, it is probably using mmap. Disable direct_io for that file type and test again.
4. Use dtruss to trace system calls. On modern macOS, you cannot trace apps in /Applications due to SIP. Copy the app to /tmp first:

   cp -R /Applications/Preview.app /tmp/Preview.app
   sudo dtruss /tmp/Preview.app/Contents/MacOS/Preview /path/to/file 2>&1

5. Remember SIP restrictions. System Integrity Protection prevents tracing system processes and accessing certain directories. If dtruss shows no output, SIP is likely blocking it. You can partially disable SIP in recovery mode, but for most debugging the "copy to /tmp" trick is sufficient.
6. Test with non-sandboxed apps first. If vim, cat, and Chrome work but Preview and TextEdit do not, the problem is sandbox-related. If nothing works, the problem is lower (permissions, mount options, or FUSE handler bugs).

The Bigger Picture

Apple's security model assumes that files come from trusted sources: the local disk, iCloud, or signed applications. Every layer of the macOS stack (sandboxing, Gatekeeper, notarization, the App Sandbox) is designed to enforce this assumption. FUSE filesystems exist outside this trust model. They are tolerated, not embraced.

macFUSE itself requires a kernel extension (or, on newer macOS versions, a system extension) that must be explicitly approved by the user in System Preferences. Each macOS update can break kernel extension loading. Apple has been slowly tightening the restrictions, making FUSE development on macOS increasingly challenging.

If you are building a FUSE filesystem that needs to work with macOS applications, these platform-specific issues require additional investigation time. What works perfectly on Linux may silently fail on macOS in ways that produce misleading error messages. The "file may be damaged" dialog tells you nothing about which layer rejected your file. You have to peel the onion one layer at a time.