Testing P2P Systems Without External Dependencies

The Problem

P2P systems are hard to test because they require two peers, a relay server, network connectivity, FUSE drivers, and specific ports. If any external dependency is unavailable, tests fail for reasons unrelated to your code.

The first version of KEIBIDROP's tests required a running relay server, and they broke constantly because the relay was down, ports were in use, or the network changed. We needed tests that worked on any machine with Go installed.

The Goal

go test ./tests/... should work on any machine. No Docker, no external relay, no manual setup. Every test creates its own universe and tears it down after.

Mock Relay

The first building block is an in-process HTTP server using Go's httptest package. It implements the same register/fetch API as the production relay:

// tests/mock_relay_test.go

type MockRelay struct {
    Server *httptest.Server
    store  map[string]string // lookupToken -> encrypted blob
    mu     sync.RWMutex
}

func NewMockRelay() *MockRelay {
    mr := &MockRelay{store: make(map[string]string)}
    mux := http.NewServeMux()
    mux.HandleFunc("POST /register", mr.handleRegister)
    mux.HandleFunc("GET /fetch", mr.handleFetch)
    mr.Server = httptest.NewServer(mux)
    return mr
}

The mock relay is functionally identical to production. Same HTTP endpoints, same Bearer token auth, same JSON format. Tests exercise the real relay protocol, not a simplified stub. It runs in-process; no Docker container, no port management, no startup delay.

TestPair Harness

The SetupPeerPair function creates two fully connected KEIBIDROP instances with a single function call:

// tests/harness_test.go

type TestPair struct {
    Alice *common.KeibiDrop
    Bob   *common.KeibiDrop
    Relay *MockRelay

    AliceSaveDir  string
    BobSaveDir    string
    AliceMountDir string
    BobMountDir   string
}

func SetupPeerPair(t *testing.T, isFuse bool) *TestPair {
    return SetupPeerPairWithTimeout(t, isFuse, 30*time.Second)
}

Each call to SetupPeerPair:

Starts a fresh mock relay
Creates temp directories (auto-cleaned by Go's testing framework)
Allocates dynamic ports in the 26100-26699 range
Generates fresh cryptographic keys for both peers
Exchanges fingerprints between Alice and Bob
Connects them via the mock relay
Returns a ready-to-use pair for file operations

A typical test is five lines: setup the pair, write a file, wait for sync, read the file, assert equality.

Dynamic Port Allocation

Tests can run in parallel. Two tests using the same ports would conflict. Ports are allocated dynamically within sub-ranges:

Alice inbound:  26100-26249
Alice outbound: 26250-26399
Bob inbound:    26400-26549
Bob outbound:   26550-26699

Each test picks a free port within its range using net.Listen on :0, then extracting the assigned port. This avoids conflicts even when running all 36 tests with parallelism.

The cgofuse Limitation

Only one FUSE mount per process on macOS. The cgofuse library registers a signal handler during mount, and two concurrent mounts race on that registration. This is a fundamental limitation of cgofuse on macOS, not a bug in our code.

// tests/harness_test.go

func SetupFUSEPeerPair(t *testing.T, timeout time.Duration) *TestPair {
    return setupPeerPairImpl(t, true, false, timeout)
    //                       Alice=FUSE  Bob=no-FUSE
}

        In production, each KEIBIDROP instance runs in its own process, so the one-mount limit
        does not apply. In tests, where two peers share a process, only one can mount FUSE. The
        workaround (Alice=FUSE, Bob=no-FUSE) actually improved test coverage by exercising both
        modes in every FUSE test.
      

Test Suite Overview

File	Tests	What it covers
integration_nofuse_test.go	8	Basic file sharing without FUSE
integration_nofuse_sync_test.go	3	Remove, rename, error types
integration_fuse_test.go	6	FUSE mount, read, write, permissions
integration_fuse_sync_test.go	5	Delete, rename, empty files, overwrite, large binary
integration_resilience_test.go	6	Health monitor, disconnect, reconnect
integration_relay_test.go	3	Relay privacy, registration, lookup
integration_error_test.go	5	Error message quality

All test files are in the tests/ directory.

Running the Tests

go test -v -count=1 -timeout 180s ./tests/...

-count=1 disables test caching. This matters for integration tests that depend on real network operations; cached results from a previous run might not reflect the current state. The 180s timeout gives room for FUSE mount/unmount operations on slower machines.

What We Learned

httptest.NewServer() runs in the same process. The test controls the server directly: inspect internal state, inject errors, shut it down mid-request. No Docker, no port management, no startup delay.

t.TempDir() handles cleanup automatically, even if the test panics. No leftover files, no manual cleanup logic.

IPv6 loopback (::1) works everywhere Go runs. It avoids IPv4 NAT complications and works identically on macOS, Linux, and Windows.

Being forced to use Alice=FUSE and Bob=no-FUSE means every test exercises both code paths. A bug in either path gets caught.

36 tests in 139 seconds is acceptable for integration tests. FUSE mount/unmount is slow (~2-3 seconds per mount on macOS) and file sync over loopback adds latency. The tradeoff between real confidence and speed is worth it.