gRPC over QUIC: faster seeks, slower bulk

We care more about surviving an IP change than about MB/s

KEIBIDROP is a peer-to-peer filesystem, so you mount a remote peer's files and read them on demand. Open a 40 GB video, seek to the middle, and only the bytes you touch cross the wire. The transport underneath is gRPC over our own encrypted TCP connection.

TCP has one problem we cannot fix from userspace, which is that the connection dies when the 5-tuple does. A laptop that moves from café Wi-Fi onto a phone hotspot changes its IP, and the transfer stops. QUIC identifies a connection by a connection ID rather than the 5-tuple, so it can migrate across an address change with the stream intact. We wanted to know whether gRPC could run over QUIC and keep that property.

We optimize for the experience of an on-demand filesystem, not for bulk throughput in MB/s. The order of priorities is: survive an IP change, keep seek latency low, stay correct, and then speed. All the numbers below are measured, and where one surprised us, we looked for the cause before believing it. The hardware is a 12-core macOS laptop, a 4-core Linux VPS, and a real link between a laptop in Barcelona and that VPS at 72 ms round trip.

gRPC runs over QUIC through a small net.Conn adapter

gRPC already speaks net.Conn, and in KEIBIDROP it runs over our encrypted SecureConn, which is a net.Conn wrapping a net.Conn, so QUIC slots in underneath the same seam. The adapter that makes one QUIC stream look like a net.Conn is small, because a quic.Stream is already a reliable, ordered byte stream with no framing to add:

type quicConn struct {
    *quic.Stream            // Read/Write/deadlines come from the stream
    conn *quic.Conn         // for the two address methods the stream lacks
}

func (c *quicConn) LocalAddr() net.Addr  { return c.conn.LocalAddr() }
func (c *quicConn) RemoteAddr() net.Addr { return c.conn.RemoteAddr() }

One detail is worth stating now, because it returns in section 5. quic.Stream.Close() is a graceful, send-only half-close, and quic-go documents that it must not be called concurrently with Write. A net.Conn.Close has to tear the whole connection down. The correct call is CancelWrite, a stream reset, not Close. We used Close first and paid for it later.

With the adapter, a listener that hands one stream up as a conn, and gRPC's NewClient pointed at a passthrough:/// target so it skips DNS, a gRPC call completes over QUIC. Unary, server-streaming, and concurrent calls all work, verified byte for byte and clean under the race detector. gRPC's own TLS stays off, and QUIC's mandatory TLS 1.3 sits underneath with a throwaway certificate, because the real confidentiality is a post-quantum handshake we add on top (ML-KEM-1024 and X25519, the same primitives KEIBIDROP already uses; QUIC's TLS 1.3 is not post-quantum and ours is).

On macOS, QUIC came out twelve times slower than TCP

The first benchmark on the macOS laptop streamed a large file both ways on loopback:

That is twelve times slower, which is large enough to want an explanation, so we profiled it. A raw QUIC stream with no gRPC gave 137 MB/s, so the gRPC layer was not the cause; the QUIC transport was. The CPU profile showed where the time went:

56.67%  syscall.rawsyscalln          # the syscall itself
33.33%  ...sendmsg   (one WriteMsgUDP per ~1400-byte packet)
23.44%  ...recvmsg
 0.6 %  crypto (AES-GCM)             # negligible

Fifty-seven percent of the time was in sendmsg and recvmsg. Crypto, which people assume is the cost of an encrypted transport, was 0.6%. What costs the time is that macOS has no UDP Generic Segmentation Offload. On Linux, quic-go hands the kernel a 64 KB buffer and the kernel splits it into packets, so one syscall sends dozens. On macOS there is no such offload, so quic-go sends one sendmsg per 1400-byte packet. At 137 MB/s that is about 100,000 send syscalls per second and as many receives.

We checked this by running the same code on the Linux VPS, which has GSO. The VPS is the slower machine (its loopback TCP is 3.4x slower than the laptop's), and its QUIC still ran at 272 MB/s against the laptop's 137. Normalized against each machine's own TCP, QUIC went from 2% on macOS to 14% on Linux. So the macOS number comes from the platform, and the design and the crypto are not what pays for it.

sendmsg_x batches the sends and lifts macOS to about 700 MB/s

macOS has neither GSO nor sendmmsg, so the syscall-batching primitives that fix this on Linux do not exist here. It does have sendmsg_x, a Darwin syscall that sends many datagrams in one call, similar to Linux sendmmsg, and it is undocumented. The syscall number is in the SDK (SYS_sendmsg_x = 481), but the struct msghdr_x it takes is not in any header. You declare it yourself from the XNU source and get the byte layout right, or the kernel reads garbage.

We wrote a small program that prints the struct offsets, checks them with a real batched send the receiver verifies, and benchmarks batch size against raw UDP send throughput:

Batching lifts the ceiling from 390 to about 700 MB/s and cuts the syscall count by 32 times. It levels off at batch 32, where the cost moves from syscall count to per-packet kernel work. So there is a real macOS lever here, about 1.8 times, with no root access, and it helps in a second way as well. The 137 MB/s QUIC transfer used about five CPU cores, all of it the per-packet work TCP hands to the kernel, and in KEIBIDROP that CPU competes with the FUSE filesystem, which is busy with encryption and chunking. Fewer syscalls per byte means fewer cores taken from the filesystem.

The primitive is now in our quic-go fork as the Darwin counterpart of the Linux GSO path, with the verified struct layout. Wiring it into a live transfer is the next section.

Send batching saves CPU, and the receive side has nothing to batch

The section above is a standalone benchmark, and wiring it into the fork's send loop, so a real QUIC transfer uses it, took more care, and it marked where batching helps and where it does not.

Two bugs came first, both on the send path. quic-go binds a dual-stack IPv6 socket, so a raw IPv4 destination is rejected, and the address has to be built as a v4-mapped IPv6 one from the socket's family. The second was subtler, because sendmsg_x can accept fewer datagrams than you hand it when the socket buffer fills mid-batch, and the first version dropped the rest. That test passed under the race detector and failed without it, the same signature as the deadlock in the next section: under the race detector the code runs slowly enough that the buffer never fills. The fix is to resend the datagrams the kernel did not take.

With that fixed the send batching works, and over a 128 MB transfer it averaged three datagrams per syscall, and the send syscall dropped from 33% of CPU to 18. That is a CPU saving and the transfer rate stays where it was, because on one machine a QUIC transfer is bound by the receive side and the per-packet work, not by the send syscall rate, so freeing send CPU does not raise the transfer rate here. It frees cores, which is what we want, because that CPU competes with the filesystem.

The receive side behaves differently, so we wired the receive counterpart, recvmsg_x, the same way. On loopback it read exactly one datagram per syscall, every time. The receiver drains each packet before the next arrives, so the socket buffer never holds more than one, and there is nothing to batch.

One idea is to add a small delay before draining, to let a burst pile up. We tried it and swept the delay:

The receive average stayed at one for every delay, and the throughput fell in proportion. Flow control is doing this, because when the receiver slows, the sender's flow-control window fills and the sender stops sending. It never gets ahead, so no burst forms, and a receiver cannot accumulate a burst in a flow-controlled protocol; the delay only throttles the connection.

To check that the batching itself works, we removed the flow control. A raw UDP flood, where the sender ignores the receiver, read 38 datagrams per syscall. So receive batching is real, but it only engages when the network delivers a burst. A real network interface does that when it coalesces interrupts; loopback does not. The receive path we ship is adaptive: it batches when the wire bursts and falls back to a plain read when it does not, so it does not pay for a batch that will not happen.

So the send batching is worth keeping as a CPU saving, the receive batching is a property of the network rather than something the receiver can force, and neither changes the seek latency from earlier. That result comes from giving the seek its own stream.

Calling Stream.Close during a Write deadlocked the teardown

The test suite passed under the race detector and hung about half the time without it. That combination points at a timing-dependent bug rather than a random flake, because the race detector slows the code down. The goroutine dump named the test, a client cancelling a server-stream, and showed Server.Stop blocked while a QUIC write stayed stuck.

Two guesses were wrong, and both were quick to disprove by measuring. Sizing the download to fit the flow-control window did not help; it still wedged. A short idle timeout had no effect, because a connection that is actively trying to send is not idle. Reading the quic-go source is what settled it in the end.

gRPC's http2Server.Close closes the underlying net.Conn, which is our quicConn.Close, which called quic.Stream.Close(), the graceful half-close from section 2, the one quic-go says must not be called concurrently with Write. gRPC's writer goroutine was concurrently blocked in a Write on that stream. The contract violation meant the write never returned, the conn never finished closing, and Server.Stop waited forever.

func (c *quicConn) Close() error {
    c.Stream.CancelRead(0)   // STOP_SENDING
    c.Stream.CancelWrite(0)  // RESET_STREAM: safe against a concurrent Write, and unblocks it
    return c.conn.CloseWithError(0, "")
}

Replacing Close with CancelWrite turned a 50% hang into ten green runs out of ten. The general point: when the race detector flips a test from pass to fail, look for the timing bug rather than adding sleeps. And a lightweight atomic counter is better than a print for diagnosing one, because the print changes the timing you are measuring. Ours made the test pass for a while and pointed us the wrong way.

A live gRPC channel moves to a new socket in mid-transfer

Migration is the reason we started, and quic-go exposes it directly: AddPath a new socket, Probe to validate the path, Switch. Getting it to fire took reading quic-go's own test. The client has to keep sending after the switch so the peer migrates too, and you must not close the old transport to prove it, because that tears the connection down.

We checked it with per-socket packet counters, moving a live gRPC channel to a new UDP socket mid-transfer:

migration: local addr 127.0.0.1:52598 -> 127.0.0.1:59126
path 1 writes +0        # old path goes silent
path 2 writes +90, reads +92   # new path carries both directions

After the switch the old path sent nothing and the new path carried both directions, and the gRPC channel kept running. On the real product this is a laptop changing networks during playback without a stall.

On real hotel Wi-Fi, TCP filled the link and QUIC did not

Loopback measures a transport's overhead, not its behaviour on a real network. So we ran it for real: a laptop on hotel Wi-Fi in Barcelona, about 45 Mbps down and 72 ms round trip, downloading from the VPS. No emulation and no offload tricks, so both TCP and QUIC send ordinary packets end to end.

A single QUIC stream dropped to about 2 Mbps of a 45 Mbps link while TCP filled it. Real Wi-Fi packet loss meets quic-go's congestion control, which backs off hard, while Linux kernel TCP handles the same loss well. We had gone in half-expecting the common claim that QUIC does better on lossy links. On a real lossy link the opposite held, and for plain bulk transfer on a bad link TCP won.

On its own this result argues against QUIC, and we nearly stopped here.

A seek under a saturating prefetch takes 109 ms on QUIC and 3 seconds on TCP

An on-demand filesystem is not used by streaming a file end to end. It is used by seeking: you open a video, jump to a position, jump again. Each jump is a small read at a random offset while a prefetch pulls bulk data in the background. The number that matters is how long that small read takes while the prefetch saturates the link.

On TCP, the prefetch and the seek share one byte stream, so the seek's bytes sit behind the prefetch already queued in the send buffer. That is head-of-line blocking, and it is the playback stutter we had seen before as self-congestion. On QUIC the seek gets its own stream, and the library interleaves its packets with the bulk, so it does not queue behind it.

We measured it on the same Barcelona link, with thirty small 16 KB reads under a saturating bulk stream:

A seek lands in about 109 ms on QUIC, roughly one round trip. On TCP it takes about three seconds, behind the prefetch. For someone scrubbing through a video the first is usable and the second is not, and the gap is larger than the bulk-throughput gap from the previous section.

This fits the earlier results together, because QUIC's advantage for us is latency isolation, a seek that does not wait behind a prefetch, together with migration, a transfer that survives a network change. For an on-demand seekable filesystem that trade is worth it. TCP can match the fast lane only with a second connection per read, which is the data-channels idea we had been circling; QUIC gives it per stream, with shared congestion control.

QUIC carries the interactive path and TCP carries the bulk

The measurements point at a specific design, in which the interactive on-demand path runs over QUIC. Seeks and prefetch on separate streams is where the seek result comes from, and migration comes with it. Plain bulk on a lossy link stays on TCP, where quic-go's congestion control loses. On fast, clean links the macOS syscall ceiling is real, and sendmsg_x batching is the lever that lifts QUIC to about 700 MB/s while freeing CPU for the filesystem. Control traffic such as heartbeats and metadata rides QUIC, since it is small and it is the channel that has to survive a network change.

None of this was obvious at the start, and several steps were the opposite of what we expected. Measuring, rather than reasoning from what QUIC is supposed to be good at, changed the answer twice: once when the slow macOS transport turned out to be a syscall cost with a fix, and once when the throughput test's verdict flipped as soon as we measured seek latency.

The experiments are small and self-contained: the gRPC-over-QUIC adapter, the sendmsg_x program, and the two WAN benchmarks.