gRPC over QUIC: faster seeks, slower bulk

The question, and what we optimize for

KEIBIDROP is a peer-to-peer filesystem. You mount a remote peer's files and read them on demand. Open a 40 GB video, seek to the middle, and only the bytes you touch cross the wire. The transport underneath is gRPC over our own encrypted TCP connection.

TCP has one problem we cannot fix from userspace. When the 5-tuple dies, the connection dies. A laptop that moves from café Wi-Fi onto a phone hotspot changes its IP, and the transfer stops. QUIC identifies a connection by a connection ID rather than the 5-tuple, so it can migrate across an address change with the stream intact. We wanted to know whether gRPC could run over QUIC and keep that property.

We optimize for the experience of an on-demand filesystem, not for bulk throughput in MB/s. The order of priorities is: survive an IP change, keep seek latency low, stay correct, and then speed. All the numbers below are measured. Where a number surprised us, we looked for the cause before believing it. The hardware is a 12-core macOS laptop, a 4-core Linux VPS, and a real link between a laptop in Barcelona and that VPS at 72 ms round trip.

Getting gRPC onto a QUIC stream

gRPC already speaks net.Conn. In KEIBIDROP it runs over our encrypted SecureConn, which is a net.Conn wrapping a net.Conn, so QUIC slots in underneath the same seam. The adapter that makes one QUIC stream look like a net.Conn is small, because a quic.Stream is already a reliable, ordered byte stream with no framing to add:

type quicConn struct {
    *quic.Stream            // Read/Write/deadlines come from the stream
    conn *quic.Conn         // for the two address methods the stream lacks
}

func (c *quicConn) LocalAddr() net.Addr  { return c.conn.LocalAddr() }
func (c *quicConn) RemoteAddr() net.Addr { return c.conn.RemoteAddr() }

One detail is worth stating now, because it returns in section 5. quic.Stream.Close() is a graceful, send-only half-close, and quic-go documents that it must not be called concurrently with Write. A net.Conn.Close has to tear the whole connection down. The correct call is CancelWrite, a stream reset, not Close. We used Close first and paid for it later.

With the adapter, a listener that hands one stream up as a conn, and gRPC's NewClient pointed at a passthrough:/// target so it skips DNS, a gRPC call completes over QUIC. Unary, server-streaming, and concurrent calls all work, verified byte for byte and clean under the race detector. gRPC's own TLS stays off. QUIC's mandatory TLS 1.3 sits underneath with a throwaway certificate, because the real confidentiality is a post-quantum handshake we add on top (ML-KEM-1024 and X25519, the same primitives KEIBIDROP already uses; QUIC's TLS 1.3 is not post-quantum and ours is).

Throughput on macOS

The first benchmark on the macOS laptop streamed a large file both ways on loopback:

Twelve times slower. That is large enough to want an explanation, so we profiled it. A raw QUIC stream with no gRPC gave 137 MB/s, so the gRPC layer was not the cause; the QUIC transport was. The CPU profile showed where the time went:

56.67%  syscall.rawsyscalln          # the syscall itself
33.33%  ...sendmsg   (one WriteMsgUDP per ~1400-byte packet)
23.44%  ...recvmsg
 0.6 %  crypto (AES-GCM)             # negligible

Fifty-seven percent of the time was in sendmsg and recvmsg. Crypto, which people assume is the cost of an encrypted transport, was 0.6 percent. The reason is that macOS has no UDP Generic Segmentation Offload. On Linux, quic-go hands the kernel a 64 KB buffer and the kernel splits it into packets, so one syscall sends dozens. On macOS there is no such offload, so quic-go sends one sendmsg per 1400-byte packet. At 137 MB/s that is about 100,000 send syscalls per second and as many receives.

We checked this by running the same code on the Linux VPS, which has GSO. The VPS is the slower machine (its loopback TCP is 3.4x slower than the laptop's), and its QUIC still ran at 272 MB/s against the laptop's 137. Normalized against each machine's own TCP, QUIC went from 2 percent on macOS to 14 percent on Linux. So the macOS number is a platform limitation, not a design or crypto cost.

Batching sends with sendmsg_x

macOS has no GSO and no sendmmsg. The syscall-batching primitives that fix this on Linux do not exist here. It does have sendmsg_x, a Darwin syscall that sends many datagrams in one call, similar to Linux sendmmsg. It is undocumented. The syscall number is in the SDK (SYS_sendmsg_x = 481), but the struct msghdr_x it takes is not in any header. You declare it yourself from the XNU source and get the byte layout right, or the kernel reads garbage.

We wrote a small program that prints the struct offsets, checks them with a real batched send the receiver verifies, and benchmarks batch size against raw UDP send throughput:

Batching lifts the ceiling from 390 to about 700 MB/s and cuts the syscall count by 32 times. It levels off at batch 32, where the cost moves from syscall count to per-packet kernel work. So there is a real macOS lever here, about 1.8 times, with no root access. It helps in a second way. The 137 MB/s QUIC transfer used about five CPU cores, all of it the per-packet work TCP hands to the kernel, and in KEIBIDROP that CPU competes with the FUSE filesystem, which is busy with encryption and chunking. Fewer syscalls per byte means fewer cores taken from the filesystem.

The primitive is now in our quic-go fork as the Darwin counterpart of the Linux GSO path, with the verified struct layout. Wiring it into a live transfer is the next section.

Wiring it in, and the receive wall

The section above is a standalone benchmark. Wiring it into the fork's send loop, so a real QUIC transfer uses it, took more care, and it marked where batching helps and where it does not.

Two bugs came first, both on the send path. quic-go binds a dual-stack IPv6 socket, so a raw IPv4 destination is rejected, and the address has to be built as a v4-mapped IPv6 one from the socket's family. The second was subtler. sendmsg_x can accept fewer datagrams than you hand it when the socket buffer fills mid-batch, and the first version dropped the rest. That test passed under the race detector and failed without it, the same signature as the deadlock in the next section: under the race detector the code runs slowly enough that the buffer never fills. The fix is to resend the datagrams the kernel did not take.

With that fixed, the send batching works. Over a 128 MB transfer it averaged three datagrams per syscall, and the send syscall dropped from 33 percent of CPU to 18. That is a CPU saving, not a throughput gain. On one machine a QUIC transfer is bound by the receive side and the per-packet work, not by the send syscall rate, so freeing send CPU does not raise the transfer rate here. It frees cores, which is what we want, because that CPU competes with the filesystem.

The receive side behaves differently. We wired the receive counterpart, recvmsg_x, the same way. On loopback it read exactly one datagram per syscall, every time. The receiver drains each packet before the next arrives, so the socket buffer never holds more than one, and there is nothing to batch.

One idea is to add a small delay before draining, to let a burst pile up. We tried it and swept the delay:

The receive average stayed at one for every delay, and the throughput fell in proportion. The cause is flow control. When the receiver slows, the sender's flow-control window fills and the sender stops sending. It never gets ahead, so no burst forms. A receiver cannot accumulate a burst in a flow-controlled protocol; the delay only throttles the connection.

To check that the batching itself works, we removed the flow control. A raw UDP flood, where the sender ignores the receiver, read 38 datagrams per syscall. So receive batching is real, but it only engages when the network delivers a burst. A real network interface does that when it coalesces interrupts; loopback does not. The receive path we ship is adaptive: it batches when the wire bursts and falls back to a plain read when it does not, so it does not pay for a batch that will not happen.

So the send batching is worth keeping as a CPU saving, the receive batching is a property of the network rather than something the receiver can force, and neither changes the seek latency from earlier. That result is stream isolation, not syscalls.

The teardown deadlock

The test suite passed under the race detector and hung about half the time without it. That combination points at a timing-dependent bug rather than a random flake, because the race detector slows the code down. The goroutine dump named the test, a client cancelling a server-stream, and showed Server.Stop blocked while a QUIC write stayed stuck.

Two guesses were wrong, and both were quick to disprove by measuring. Sizing the download to fit the flow-control window did not help; it still wedged. A short idle timeout had no effect, because a connection that is actively trying to send is not idle. Reading the source settled it.

gRPC's http2Server.Close closes the underlying net.Conn, which is our quicConn.Close, which called quic.Stream.Close(), the graceful half-close from section 2, the one quic-go says must not be called concurrently with Write. gRPC's writer goroutine was concurrently blocked in a Write on that stream. The contract violation meant the write never returned, the conn never finished closing, and Server.Stop waited forever.

func (c *quicConn) Close() error {
    c.Stream.CancelRead(0)   // STOP_SENDING
    c.Stream.CancelWrite(0)  // RESET_STREAM: safe against a concurrent Write, and unblocks it
    return c.conn.CloseWithError(0, "")
}

Replacing Close with CancelWrite turned a 50 percent hang into ten green runs out of ten. The general point: when the race detector flips a test from pass to fail, look for the timing bug rather than adding sleeps. And a lightweight atomic counter is better than a print for diagnosing one, because the print changes the timing you are measuring. Ours made the test pass for a while and pointed us the wrong way.

Migration

Migration is the reason we started. quic-go exposes it directly: AddPath a new socket, Probe to validate the path, Switch. Getting it to fire took reading quic-go's own test. The client has to keep sending after the switch so the peer migrates too, and you must not close the old transport to prove it, because that tears the connection down.

We checked it with per-socket packet counters, moving a live gRPC channel to a new UDP socket mid-transfer:

migration: local addr 127.0.0.1:52598 -> 127.0.0.1:59126
path 1 writes +0        # old path goes silent
path 2 writes +90, reads +92   # new path carries both directions

After the switch the old path sent nothing and the new path carried both directions. The gRPC channel kept running. On the real product this is a laptop changing networks during playback without a stall.

The real WAN test

Loopback measures a transport's overhead, not its behaviour on a real network. So we ran it for real: a laptop on hotel Wi-Fi in Barcelona, about 45 Mbps down and 72 ms round trip, downloading from the VPS. No emulation and no offload tricks, so both TCP and QUIC send ordinary packets end to end.

A single QUIC stream dropped to about 2 Mbps of a 45 Mbps link while TCP filled it. The cause is real Wi-Fi packet loss and quic-go's congestion control, which backs off hard, while Linux kernel TCP handles the same loss well. We had gone in half-expecting the common claim that QUIC does better on lossy links. On a real lossy link the opposite held. For plain bulk transfer on a bad link, TCP won.

On its own this result argues against QUIC. We nearly stopped here.

Seek latency under prefetch

An on-demand filesystem is not used by streaming a file end to end. It is used by seeking. You open a video, jump to a position, jump again. Each jump is a small read at a random offset while a prefetch pulls bulk data in the background. The number that matters is how long that small read takes while the prefetch saturates the link.

On TCP, the prefetch and the seek share one byte stream, so the seek's bytes sit behind the prefetch already queued in the send buffer. That is head-of-line blocking, and it is the playback stutter we had seen before as self-congestion. On QUIC the seek gets its own stream, and the library interleaves its packets with the bulk, so it does not queue behind it.

We measured it on the same Barcelona link. Thirty small 16 KB reads under a saturating bulk stream:

A seek lands in about 109 ms on QUIC, roughly one round trip. On TCP it takes about three seconds, behind the prefetch. For someone scrubbing through a video the first is usable and the second is not, and the gap is larger than the bulk-throughput gap from the previous section.

This fits the earlier results together. QUIC's advantage for us is latency isolation, a seek that does not wait behind a prefetch, together with migration, a transfer that survives a network change. For an on-demand seekable filesystem that trade is worth it. TCP can match the fast lane only with a second connection per read, which is the data-channels idea we had been circling; QUIC gives it per stream, with shared congestion control.

What we build

The measurements point at a specific design. The interactive on-demand path runs over QUIC. Seeks and prefetch on separate streams is where the seek result comes from, and migration comes with it. Plain bulk on a lossy link stays on TCP, where quic-go's congestion control loses. On fast, clean links the macOS syscall ceiling is real, and sendmsg_x batching is the lever that lifts QUIC to about 700 MB/s while freeing CPU for the filesystem. Control traffic such as heartbeats and metadata rides QUIC, since it is small and it is the channel that has to survive a network change.

None of this was obvious at the start, and several steps were the opposite of what we expected. Measuring, rather than reasoning from what QUIC is supposed to be good at, changed the answer twice: once when the slow macOS transport turned out to be a syscall cost with a fix, and once when the throughput test's verdict flipped as soon as we measured seek latency.

The experiments are small and self-contained: the gRPC-over-QUIC adapter, the sendmsg_x program, and the two WAN benchmarks.