Rekey While the Video Plays: Key Rotation Across TCP and QUIC

Where This Starts

An earlier post described how KEIBIDROP rotates session keys after 1 GB or one million messages: the initiator generated fresh seeds, encapsulated them with X25519 and ML-KEM, and swapped keys over the session's RPC channel. Writing the 0.4.0 security notes forced us to be precise about what that gave us. The seeds were fresh, but they traveled encapsulated to the peers' long-term identity keys, so a rotation limited the damage of a leaked session key and nothing more. An attacker who recorded the exchange and stole the identity keys later could decapsulate every seed and re-derive every key. It also coordinated each rotation through a request and response, which is a round trip in the middle of whatever the connection was doing.

For 0.4.0 we replaced it with an in-band ratchet: each direction of each connection advances its own key without any round trip, and the rotation rides inside the data stream itself. At the same time, 0.4.0 grew a second transport lane, a QUIC channel over UDP that carries control traffic, metadata, and on-demand reads while bulk transfer stays on TCP. Two mechanisms and two lanes had to meet, and this post is about what that took, what it cost in latency (measured: nothing), and the two bugs the integration found on the way.

The In-Band Ratchet

The 12-byte AEAD nonce now has three parts:

[4-byte direction prefix | 2-byte key epoch | 6-byte counter]

Every direction keeps a chain key. When a direction crosses a threshold (1 GB written, one million messages, or 60 seconds since its last rotation, whichever comes first), the writer derives the next chain key with a one-way KDF, discards the old one, bumps the epoch in the nonce, and seals the very next frame under the new key. The check runs at the top of the write path, before sealing, so the frame that crosses the threshold is already encrypted under the fresh key. The receiver sees the epoch increment in the nonce and follows with the same derivation. There is no negotiation, no pause, and no message exchange: the rotation is a local computation on both ends, a few microseconds of HKDF.

Because the old chain key is destroyed at each step, a leaked live key exposes at most one epoch of traffic, bounded by the thresholds above. And because the derivation check runs before sealing, an idle connection rotates on its first write after the cadence elapses, so no frame is ever sealed under a key older than the thresholds allow.

Two Lanes, One Gap

The capability is negotiated in the handshake, and the negotiation result was applied to the TCP socket pair. The QUIC lane's connections are built elsewhere, from separate per-direction keys derived in the same handshake, and they never received the switch. They would have sat at epoch 0 forever, encrypting everything under their handshake keys while the TCP pair rotated next to them. The fix is one call at each of the two places a QUIC connection is wrapped, gated on the same negotiated capability. Small fix, but it only exists because we went looking for which connections the negotiation actually reached.

The Numbers

All of this is measured on one machine over loopback, deliberately. Loopback removes network noise, so whatever latency the rotation itself adds is the only thing left to see. The rotation threshold is forced down to 1 MiB so a modest stream crosses many epochs; at the production threshold of 1 GB you would need terabytes to see this many rotations.

First, a 64 MiB on-demand stream in 256 KiB reads, the same path a video player uses. The serving side rotated its key 63 times during the stream. Reads that landed on a rotation are compared against reads that did not:

The two distributions are the same within noise. The slowest read in the entire run was 4.2 ms. A paced playback test tells the same story from the user's side: a 32 MiB clip played at Blu-ray bitrate (40 Mbps) with a 2 second jitter buffer crossed 9 rotations with zero underruns; the slowest fetch was 171 ms against the 2000 ms budget. Bulk is unaffected too: pulling 128 MiB with rotation forced every 1 MiB (97 rotations) lands within noise of the same pull with rotation off, ratios of 1.06 and 1.03 across runs.

The QUIC lane gets its own proof at the unit level: 2 MiB echoed through the real QUIC channel with a 64 KiB threshold advanced 31 epochs in each direction, byte-perfect, with a worst chunk round trip of 1.04 ms. A soak test then runs 40 rekey rounds back to back and checks that the goroutine count stays flat (70 before, 70 after), so nothing leaks per rotation.

The Test That Reported Zero Rekeys

The integration was not clean on the first pass, and the failure is the most instructive part. The streaming tests above initially failed on the merged branch with "expected rekeys during playback, got 0". The same tests passed on the rekey branch alone. The natural conclusion would be that the merge broke the ratchet.

It had not. On the merged branch, on-demand reads ride the QUIC lane, and the QUIC lane was rotating exactly as designed. The tests were watching epochs through the health monitor, which tracks the TCP socket pair only. The data had moved to a lane the instrument could not see, so 63 rotations looked like zero. We confirmed it by running the same tests against the rekey branch in a separate worktree (data on TCP: pass) and against the merged branch after adding a QUIC epoch accessor and widening the test's observer to the maximum epoch across all lanes of both peers (pass, 63 rotations counted).

The lesson generalizes. When traffic can take more than one path, any assertion of the form "X happened during the transfer" has to observe every path, or a routing change will turn a passing test into a lie in either direction. The epoch accessor stayed in the product as a diagnostic, and the same widened view now feeds the near-wrap rescue described below.

What the Race Detector Found

The rekey branch also brought a set of teardown-race regression tests, and under the race detector one of them flagged the QUIC code we were merging into: four places read the session pointer without the lock that teardown takes when it clears that pointer. Three were on request paths; one was in the background loop that redials the QUIC channel, which can genuinely fire while a session is being torn down. All four now snapshot the pointer under the lock. The rekey work did not cause these, but its tests found them, which is a good argument for merging test suites early instead of cherry-picking code.

One more measurement lesson from the same session: the bulk throughput test failed once with a ratio of 0.57, rotation on against rotation off. The cause was not the code. Two throughput suites were running on the same machine at the same time, one of them our own comparison run on the second branch. Isolated re-runs gave 1.06 and 1.03. Loopback throughput tests measure the machine as much as the code, and they only mean something on an idle box.

Fresh Entropy for the QUIC Lane

The ratchet protects against a leak of the live key, but it is deterministic: all epochs derive from the handshake, and the handshake encapsulates to long-term identity keys. An attacker who records traffic today and steals the identity keys later can replay the whole chain. This is the same weakness the old rekey had, and rotating faster does not fix it; what fixes it is entropy that never touches a long-term key. So 0.4.0 adds the fold: right after connecting, the peers run one ephemeral ML-KEM-1024 + X25519 exchange (about 1.6 KB each way, one round trip), where the key pairs are generated for that single round, the result is mixed into the ratchet, and the ephemeral private keys are destroyed. Traffic recorded after the fold stays sealed even if every long-term key later leaks; the handful of frames sent before a direction folds, bounded by about one heartbeat interval, are protected only by the handshake keys. This is the construction iMessage PQ3 and Signal use, applied per connection.

While writing the security documentation for the release we found the fold had the same shape of gap the ratchet had: it staged its secret on the TCP pair only, and the lane that carries on-demand reads is QUIC. So the fold now stages the same secret on every live connection, and the initiating peer re-runs a round whenever a QUIC connection comes up, since the channel dials in the background and can appear after the first fold finished. Repeat rounds are safe because staging is idempotent and a newer round supersedes an unapplied one.

The proof test pins the byte threshold at 2⁶² so an ordinary rotation is impossible, runs one fold round, and then checks that both ends of the real QUIC channel advanced to epoch 1 anyway. The only way that epoch can move in that test is the folded rotation, and the echoed data across it is verified byte for byte.

One design note worth recording: we considered sending the fold exchange over both lanes at once for resilience, and rejected it. KEM encapsulation is randomized, so answering the same request twice derives two different secrets, and the peers could commit different ones. The safe version is failover: the round tries the QUIC channel first and falls back to TCP, one round in flight at a time. True send-on-both would need a round identifier and a response cache on the responder, which the protocol has room for but does not need yet.

The Epoch Budget

The epoch field is 16 bits, and the counter design cannot survive a wrap, so the writer stops rotating a few epochs short of the ceiling and holds. That budget is larger than it sounds: about 65,500 rotations per direction, which at the production threshold is 64 TB through one connection, or 45 days of rotating once a minute on the idle cadence. Before a lane gets near the ceiling, the health monitor forces one ordinary re-handshake during an idle moment, which re-derives every key and resets every lane to epoch 0. The monitor watches the QUIC lane's epochs too, through the same accessor the tests use. If the rescue somehow never ran, the connection would keep working with rotation paused, degraded rather than broken.

What This Means If You Use KeibiDrop

From 0.4.0 on, key rotation is continuous and invisible. Stream a movie from a peer and the keys underneath rotate on schedule without a stutter; leave a session idle overnight and the first write in the morning is sealed under a fresh key; record the traffic and steal the identity keys later and the fold has already made that recording worthless, on both lanes. None of this asks anything of the user, which was the entire point.

The honest cost: 0.4.0 cannot talk to 0.3.x. The negotiated capability is bound into key derivation on purpose, so a middlebox that strips the bit kills the handshake instead of silently downgrading the session, and that same binding changes the derived keys for everyone. Both ends of a pair need the update. A session where the peer genuinely lacks the ratchet still works at epoch 0, and it now logs a loud warning so the downgrade cannot pass unnoticed.

Status

The rekey and QUIC packages pass their unit, race-detector, and integration suites on macOS, Linux, and Windows. What remains before we call the release done: a soak over a real WAN link rather than loopback, and an external pair of eyes on the crypto, which no amount of our own testing replaces. The measured numbers in this post come from the session log we keep for every benchmark run, raw output included, so future regressions have something exact to be compared against.