b5a7a3e. Findings: 30 to 38 MB/s into the Windows host and 13 to 25 MB/s out; connect time about 6.3 s, dominated by an IPv6 dial timeout before bridge fallback; FUSE first byte under 1.5 s with sequential on-demand read near 1 MB/s. The throughput ceiling is the bandwidth-delay product, not link capacity or CPU. FUSE mode did not start on the Windows host on this build; the fix has since merged as PR #181, and a follow-up change (PR #182) made on-demand reads 10 to 17 times faster, from about 2 MB/s up to 22 to 30 MB/s, by fetching a 16 MiB block per round trip instead of 512 KiB, so they now match whole-file transfer.
Setup
Three machines, two pairings. One end is always the Singapore box.
| SG | Singapore, Windows Server, 4 vCPU / 8 GB, WinFsp installed. IPv4 only. |
| VPS | Timisoara, Linux. Also runs the relay and the TCP bridge. |
| laptop | Iasi, Linux, residential link 500 to 600 Mbps wired (62 to 75 MB/s). |
| Pairing A | VPS to SG, about 200 ms RTT, datacenter to datacenter. |
| Pairing B | laptop to SG, about 330 ms RTT, home to datacenter. |
Method: each cell is the median across repeated trials, with the trial count n shown in every table: 5 at 10 and 100 MB, 3 at 1 GB, 1 at 5 and 10 GB for bulk; 10 for handshake; 3 for FUSE. All three peers run the same commit. X to Y means X sends and Y receives. Bulk transfer uses kd add then a timed pull, no FUSE in the path.
Bulk throughput
Pairing A, VPS to SG, about 200 ms
| Transfer | Size | n | Median MB/s | p95 |
|---|---|---|---|---|
| SG to VPS | 10 MB | 5 | 25.6 | 28.0 |
| VPS to SG | 100 MB | 5 | 35.5 | 41.5 |
| VPS to SG | 1 GB | 3 | 37.6 | 40.4 |
| SG to VPS | 1 GB | 3 | 13.2 | 13.6 |
| VPS to SG | 5 GB | 1 | 34.6 | 34.6 |
| SG to VPS | 5 GB | 1 | 8.6 | 8.6 |
| VPS to SG | 10 GB | 1 | 18.8 | 18.8 |
| SG to VPS | 10 GB | 1 | 12.9 | 12.9 |
Pairing B, laptop to SG, about 330 ms
| Transfer | Size | n | Median MB/s | p95 |
|---|---|---|---|---|
| SG to laptop | 10 MB | 5 | 20.4 | 21.8 |
| laptop to SG | 100 MB | 5 | 29.6 | 30.5 |
| laptop to SG | 1 GB | 3 | 29.0 | 29.3 |
| SG to laptop | 1 GB | 3 | 13.4 | 15.1 |
| laptop to SG | 5 GB | 1 | 32.9 | 32.9 |
| laptop to SG | 10 GB | 1 | 31.3 | 31.3 |
Handshake
Time to connect to the Singapore box, parsed from the joiner's own log (relay lookup to connected).
| Connects to SG from | n | Median ms | p95 | min | max |
|---|---|---|---|---|---|
| VPS (pairing A) | 10 | 6316 | 6473 | 6094 | 6476 |
| laptop (pairing B) | 10 | 6220 | 6481 | 5977 | 6507 |
FUSE on-demand
A Linux peer mounts the virtual drive and reads a file hosted on SG. Content streams on demand over the link. ttfb is the time to the first byte (the first 512 KB chunk). The mount is Linux because the Windows mount has a bug, covered below.
| Mount reads from SG | Size | n | ttfb median ms | p95 | read MB/s |
|---|---|---|---|---|---|
| VPS mount (pairing A) | 10 MB | 3 | 632 | 1242 | 0.9 |
| VPS mount (pairing A) | 100 MB | 3 | 509 | 635 | 1.0 |
| laptop mount (pairing B) | 10 MB | 3 | 709 | 1423 | 0.8 |
| laptop mount (pairing B) | 100 MB | 3 | 482 | 549 | 0.8 |
What the numbers say
The limit is round-trip time, not bandwidth and not CPU. The home link is 500 to 600 Mbps, which is 62 to 75 MB/s, and the best run used about half of it. The box is a 4 vCPU machine that stayed near idle, and ChaCha20 runs at gigabytes per second per core, so encryption is not the wall either. The wall is the bandwidth delay product: one gRPC stream with a 16 MiB window cannot keep a 330 ms pipe full, because filling it needs roughly 20 to 25 MB in flight at the home link rate (500 to 600 Mbps over a 330 ms round trip). A bigger window or parallel pull streams would close that gap.
The Singapore box sends at about half the rate it receives. At 1 GB on pairing A it pulls in 37.6 MB/s but pushes out 13.2 MB/s, and the same shape holds across sizes and on pairing B. The receive path is fine. The send path is where the work is.
The 6.3 second connect is one timeout. The box has no IPv6, so the direct IPv6 dial always fails and waits out its deadline before the bridge takes over. Skipping the direct dial when a peer advertises no IPv6 removes most of that. The same change helps phones, which are usually IPv4 only and hit the same wait.
On-demand FUSE opens fast: the first byte comes back in half a second to a second and a half, which is fine for opening a document, scrubbing a video, or reading part of a dataset. But reading a whole large file straight through settled near 1 MB/s, because each 512 KB chunk cost a full round trip at this latency. That was the starting point. The next section is what we did about it.
Making on-demand reads fast
Once FUSE mounted on Windows (that was its own fix, see below), the first thing worth measuring was an on-demand read: open a file on the mount and read it straight through while it streams from the peer. It was slow, about 2 MB/s, and it stayed about 2 MB/s no matter the file size. Copying the same file with a plain bulk transfer ran at 30 MB/s. So reading through the mount was more than ten times slower than just fetching the file, which is backwards from what you want.
The reason is round trips. An on-demand read asked the peer for one 512 KiB chunk, waited for it to come back, returned it, then asked for the next one. Over a link with 200 to 330 ms of round-trip time, that wait is the whole story: almost all the time goes into waiting for the network, not into moving bytes. One round trip per 512 KiB works out to 512 KiB divided by the round trip, which lands right on the 2 MB/s we saw, and explains why the file size did not matter.
The fix is to stop asking for 512 KiB at a time. When a read misses the cache, fetch a whole 16 MiB block in one round trip, write it to the local cache, and serve the reads that follow from disk. 16 MiB is the size of the gRPC stream buffer, so the block still arrives in a single message and does not get split. One round trip now covers 16 MiB instead of 512 KiB, so a sequential read pays the network once per 16 MiB and reads everything in between at disk speed. This is not a Windows fix. It lives in the shared read path, and 16 MiB is the transfer window, so the same change speeds up macOS and Linux mounts too. We measured it on the Windows box because that is the far, high-latency peer, but the round-trip problem and the fix are the same on every platform. The change is PR #182.
Here is the same cold on-demand read, before and after:
| Pairing | File | Before MB/s | After MB/s | Speedup |
|---|---|---|---|---|
| vps-sg (~200 ms) | 100 MB | 2.05 | 30.4 | 15x |
| vps-sg (~200 ms) | 1 GB | ~2.3 | 22.6 | 10x |
| local-sg (~330 ms) | 100 MB | 1.65 | 28.8 | 17x |
| local-sg (~330 ms) | 1 GB | ~1.6 | 22.6 | 14x |
On a 100 MB or 1 GB file, an on-demand read went from about 2 MB/s to 22 to 30 MB/s, and it now tracks the whole-file transfer ceiling. The read path stopped being the bottleneck; the network is. (The 1 GB before figure is estimated from the per-chunk rate. A real 1 GB read at 2 MB/s takes seven to twelve minutes, so we did not sit through one.)
A small file is the easy case. Anything up to 16 MiB fits in a single block, so a 10 MB file comes down in one round trip and the rest of the read is local disk. That is why a small file feels instant after the first touch. It is the file already being on disk rather than a sustained network rate, so we do not quote a headline number for it.
The one tradeoff is reading a little out of a lot of large files. Jumping to one spot in a 1 GB file now pulls the surrounding 16 MiB rather than 512 KiB. For reading files start to finish, and for a folder of many small files, that is exactly what you want. For scattered small reads across huge files it fetches more than it uses, which is a knob we can tune later.
Update (June 2026): fast turned out not to be the same as smooth. A full-speed read hit 22 to 30 MB/s, but a video player reads at its bitrate, not at full speed, so it kept freezing for a round trip at every 16 MiB boundary. A follow-up adds predictive read-ahead that keeps the next blocks buffered ahead of the playhead, which took a cold 3 MB/s paced playback from five freezes to zero on the Windows mount: Smooth On-Demand Playback Over 200 ms.
Defects identified
FUSE mode did not start on the Windows host on this build. host.Mount returned false and the daemon fell back to no-FUSE: kd list showed the files, but the mount point stayed an empty folder and neither local nor remote files surfaced. The cause is the mount point being created before WinFsp runs. WinFsp takes the mount point as a drive letter (for example K:), an auto-assigned drive (*), or a directory it creates itself, and it rejects a path that already exists. PR #181 (now merged, commit a8da24a) removes the pre-creation on Windows. No-FUSE transfer is unaffected, which is why the FUSE tables above use Linux mounts.
One 10 GB transfer looked like it stalled at 12 MB. A re-run moved 3.7 GB in 4 minutes at a steady 15.6 MB/s, so it was a blip on the residential link, not a defect. Not filed.
Planned changes
Skip the direct dial when a peer advertises no IPv6. This removes most of the 6.3 s connect time on the common case and applies to mobile, which is also IPv4 only.
Add a push-based send to remove the asymmetry where the host sends at half its receive rate.
For bulk throughput, raise the window or pull with several streams. Tradeoff: saturating the link degrades on-demand read latency, so this stays opt-in for bulk transfers rather than the default for the interactive mount.
Medians over the per-table trial counts (n), all peers on commit b5a7a3e. Pairing A is VPS to SG at about 200 ms. Pairing B is laptop to SG at about 330 ms. All traffic over the TCP bridge.