A 0.26 s seek on an idle WAN link takes up to 4.7 s on a full one

We measured a Windows server in Singapore reading a Linux box in Timisoara

The Linux box shares a folder. The Windows server mounts it through WinFsp and reads it with a small program that opens a file, jumps to 48 offsets in a fixed random order, reads 2 MiB at each, and times every read. Each pass reads a file nobody has read before, so nothing comes from a local cache. A TCP connect from the Singapore server to our bridge in Timisoara took 180 to 201 ms, n=3.

The load is a second program on the same server reading a fresh 2 GB file front to back in 1 MiB reads. It starts 10 s before the jumps and outlives them. It ran at 23 to 28 MB/s on the direct path and 18 to 22 MB/s through the bridge, which is what this pair of machines moves in that direction.

Two topologies. In the first the Windows side dials the Linux box's public address and the data flows on a direct TCP connection. In the second both peers go through the bridge, which is the path two machines behind home routers take. Two trials per cell, the 0.4.9 release candidate and the v0.4.8 release, read-ahead and prefetch at their defaults.

An idle link costs one round trip and one block per jump

With nothing else on the link a jump took 0.24 to 0.29 s at the median and 0.27 to 0.32 s at the 95th percentile, across the four idle passes on the release candidate. The arithmetic is one round trip of about 200 ms plus about 80 ms to move a 2 MiB block at 25 MB/s. A jump is one request and one block, whatever the file size.

The first open of a file in a session is the exception: 2.0 to 2.1 s in each of the four sessions. Later fresh files opened in 0.6 to 1.0 s. That cost sits in the open, before the first read, and this measurement did not separate what it is made of.

Under a full link the median holds on a direct connection and the tail triples

With the 2 GB read filling the direct connection, the median jump went from 0.26 s to 0.28 s. The 95th percentile went from 0.27 s to 0.68 and 0.81 s in the two trials, and the slowest jump took 0.71 and 1.02 s. A demand fetch for a jump joins the same queue as the read-ahead blocks of the sequential reader, and every so often it lands behind a full window of them.

The sequential reader felt the same queue from the other side. Its 1 MiB reads took 1 ms at the median because read-ahead had the bytes waiting, 0.5 s at the 95th percentile, and 3.3 to 4.1 s at worst, on both builds and in both topologies.

Through the relay the median nearly doubles and the slowest jump took 4.7 s

The same passes through the bridge: the idle median was 0.25 to 0.29 s, close to direct, since the bridge sits on the Linux box's side of the link. Under load the median went to 0.48 s in both trials, the 95th percentile to 0.69 and 1.83 s, and the slowest jump to 0.80 and 4.70 s. Both legs of the session cross one relay process and its queue, on top of the link's own.

Release candidate, Singapore Windows Server 2022 reading a Timisoara Linux VPS over a 200 ms link, 48 jumps of 2 MiB into a fresh 96 MB file, two trials per row. The v0.4.8 release measured the same within that noise; its idle passes carried two single-jump outliers of 5.0 and 10.9 s that two trials cannot attribute to the build or to the link.

Switching the control lane off changed nothing at this block size

The relay path carries a second lane over UDP for small urgent reads, so an open or a short read does not queue behind bulk data on one TCP connection. In July that lane took a 16 KiB read under a saturating transfer from 1.17 s to 0.18 s at the median on this same 200 ms pairing, 30 reads per cell. We repeated the bridge passes with the lane switched off. The loaded median was 0.26 and 0.47 s, the loaded 95th percentile 0.62 and 1.02 s, and the slowest jump 0.78 and 1.38 s. Two trials each cannot tell those rows from the lane-up rows above.

A 2 MiB block is 128 times the July read. Whichever lane carries the request, the block crosses the same full link, and at 20 MB/s that is 100 ms of transfer behind whatever is already in flight. The lane's measured benefit is the small read: the open, the listing, the first bytes of a file. The block behind a jump is bandwidth, and a full link has none to spare.

On a dual-stack host the control lane dialed IPv6 and never came up

The direct sessions above ran without the control lane, and the log said why. The Linux box has a public IPv6 address and a public IPv4 address, and advertises both. The TCP dial walks the list: IPv6 first, no route from the Singapore server, then IPv4, which connects. The UDP lane dials one address, the IPv6 one, five times, then retries every 30 s for the life of the session:

QUIC control dial attempt failed  attempt=4  error="quic control dial [2a06:...]:26431: no route to host"
QUIC control dial burst exhausted
QUIC channel unavailable; staying on TCP, re-probing periodically  every=30s

The session works, every byte arrives, and nothing on screen says the lane is missing. The same lines sit in the v0.4.8 logs. The fix is one line: dial the lane at the address family that answered the TCP dial. It is queued behind this release, because it changes the connect path and we test that on real machines before it ships.

A heartbeat that shared its cancel with Stop died at the first disconnect

A saved contact posts a heartbeat to the relay while it runs, and the relay keeps that presence for 60 s. The desktop app started its heartbeat on the process context and handed the same cancel to Stop. The first disconnect of a run called Stop, the heartbeat stopped with it, and the app kept running with the relay reading it as absent until the next restart. Every desktop build up to v0.4.8 has this.

The release candidate reads presence to skip a paid relay leg for a contact that is not there. Against an old app with a dead heartbeat that skip is wrong, and the handshake carries no version field to tell the builds apart. It does carry a new field, the cipher a peer committed, which only fixed builds send. That field is now the build marker: presence decides anything only for peers whose last handshake declared a cipher, and it retires itself as the old builds go away. A capability field in the handshake from day one would have made this a one-line check instead of a proxy.

gRPC waits two hours before its first keepalive ping

The browser build of KEIBIDROP reaches a desktop through a WebSocket hop that drops a connection after 120 s of silence. A browser sends no keepalive pings of its own, and the gRPC server default on the desktop side is a first ping after two hours. An idle browser session therefore died at the hop. The desktop's servers now ping an idle client every 45 s with a 60 s timeout, and accept client pings 30 s apart or slower, so a browser that adds its own pings later is not cut off for pinging too often. A ping goes out only after 45 s with nothing read, so a transfer never pings and connect time does not change.

The on-demand cache on Windows cannot be dropped while the mount is up

Our first pass under load came back at 391 MB/s. That rate is a local disk. The mount keeps the cache file of every file it has served open, and Windows refuses to delete an open file. The clear between passes had silently done nothing and the second pass read the first pass's bytes. Every pass now reads a file nobody has read before. If you benchmark a mount on Windows, count the bytes on the wire as well as the timings.