Testing P2P Systems Without External Dependencies

Testing P2P usually needs two peers, a relay, FUSE and free ports

P2P systems are hard to test because they require two peers, a relay server, network connectivity, FUSE drivers, and specific ports. If any external dependency is unavailable, tests fail for reasons unrelated to your code.

The first version of KEIBIDROP's tests required a running relay server, and they broke constantly because the relay was down, ports were in use, or the network changed. We needed tests that worked on any machine with Go installed.

The suite has to run on any machine with no setup at all

Running go test ./tests/... should work on any machine, with no Docker, no external relay and no manual setup, because every test creates its own universe and tears it down after.

The mock relay is an in-process httptest server speaking the real API

The first building block is an in-process HTTP server using Go's httptest package. It implements the same register/fetch API as the production relay:

// tests/mock_relay_test.go

type MockRelay struct {
    Server *httptest.Server
    store  map[string]string // lookupToken -> encrypted blob
    mu     sync.RWMutex
}

func NewMockRelay() *MockRelay {
    mr := &MockRelay{store: make(map[string]string)}
    mux := http.NewServeMux()
    mux.HandleFunc("POST /register", mr.handleRegister)
    mux.HandleFunc("GET /fetch", mr.handleFetch)
    mr.Server = httptest.NewServer(mux)
    return mr
}

The mock relay is functionally identical to production. Same HTTP endpoints, same Bearer token auth, same JSON format. Tests exercise the real relay protocol, not a simplified stub. It runs in-process; no Docker container, no port management, no startup delay.

One call builds two connected peers

The SetupPeerPair function creates two fully connected KEIBIDROP instances with a single function call:

// tests/harness_test.go

type TestPair struct {
    Alice *common.KeibiDrop
    Bob   *common.KeibiDrop
    Relay *MockRelay

    AliceSaveDir  string
    BobSaveDir    string
    AliceMountDir string
    BobMountDir   string
}

func SetupPeerPair(t *testing.T, isFuse bool) *TestPair {
    return SetupPeerPairWithTimeout(t, isFuse, 30*time.Second)
}

Each call to SetupPeerPair:

1. Starts a fresh mock relay
2. Creates temp directories (auto-cleaned by Go's testing framework)
3. Allocates dynamic ports in the 26100-26699 range
4. Generates fresh cryptographic keys for both peers
5. Exchanges fingerprints between Alice and Bob
6. Connects them via the mock relay
7. Returns a ready-to-use pair for file operations

A typical test is five lines: setup the pair, write a file, wait for sync, read the file, assert equality.

Each peer gets its own port range so tests can run in parallel

Tests can run in parallel, and two of them using the same ports would conflict, so ports are allocated dynamically within sub-ranges:

Alice inbound:  26100-26249
Alice outbound: 26250-26399
Bob inbound:    26400-26549
Bob outbound:   26550-26699

Each test picks a free port within its range using net.Listen on :0, then extracting the assigned port. This avoids conflicts even when running all 36 tests with parallelism.

macOS allows one FUSE mount per process

Only one FUSE mount per process on macOS. The cgofuse library registers a signal handler during mount, and two concurrent mounts race on that registration. This is a fundamental limitation of cgofuse on macOS, not a bug in our code.

// tests/harness_test.go

func SetupFUSEPeerPair(t *testing.T, timeout time.Duration) *TestPair {
    return setupPeerPairImpl(t, true, false, timeout)
    //                       Alice=FUSE  Bob=no-FUSE
}

This is what each test file covers

All test files are in the tests/ directory.

Run them with -count=1, because caching hides real failures

go test -v -count=1 -timeout 180s ./tests/...

-count=1 turns off test caching, which matters for integration tests that depend on real network operations; cached results from a previous run might not reflect the current state. The 180s timeout gives room for FUSE mount/unmount operations on slower machines.

An in-process server lets the test inspect and break the server

httptest.NewServer() runs in the same process. The test controls the server directly: inspect internal state, inject errors, shut it down mid-request, with no Docker, no port management and no startup delay.

t.TempDir() handles cleanup automatically even if the test panics, so there are no leftover files and no manual cleanup logic.

IPv6 loopback (::1) works everywhere Go runs. It avoids IPv4 NAT complications and works identically on macOS, Linux, and Windows.

Being forced to use Alice=FUSE and Bob=no-FUSE means every test exercises both code paths. A bug in either path gets caught.

36 tests in 139 seconds is acceptable for integration tests. FUSE mount/unmount is slow (~2-3 seconds per mount on macOS) and file sync over loopback adds latency. The tradeoff between real confidence and speed is worth it.