Benchmark results

Most numbers here are local reference numbers, not a capacity claim. Unless a section says otherwise, the server and the load generator ran on the same laptop, over loopback. Those sections show where the cost goes and that an optimisation helped; they say nothing about a 2 vCPU / 4 GB server or a real network. The section On a free cloud VM, over the internet is the exception: there the generator ran on another machine. Everything not listed is not run.

Generated by bench/make_results_doc.py from the raw JSON in bench/results/. Method and caveats: bench/README.md.

#Environment

date:      2026-09-29T12:40:50Z
os:        Darwin 25.5.0 arm64
cpu:       Apple M1 Pro (10 logical cores)
ram:       16 GB
go:        go version go1.26.5 darwin/arm64
ulimit -n: 1048575
git:       no commit

Server and load generator on one host (loopback), Google Chrome not involved. Media disabled (no LiveKit). Each run is a single 45 s window after an 8 s warm-up: run-to-run variance was not measured, so treat differences below ~20 % as noise.

#Method (scenario A and E)

• Bots join over HTTP, connect a WebSocket and walk BFS paths between random targets, spending ~30 % of the time walking, like people who stop to talk.
• distributed: targets anywhere on the 60x36-tile starter map (interest areas still cover most of it); concentrated: everybody inside the 17x13-tile reception, the worst case for fan-out.
• Propagation latency: probe bots change direction and another bot in the same process stamps when the server's frame shows the change (one clock, includes the generator's own scheduling).
• Tick percentiles come from a coarse histogram (upper bucket bound); mean and max are exact.

#Before and after "state-change records"

Baseline = every moving player's position sent every tick to every observer (decision 0006). Records/s per client is how many entity records each client received per second.

#Full table after the change

#Browser under a crowd (real Chrome)

One real Chrome page (headless, GPU through ANGLE (Apple, ANGLE Metal Renderer: Apple M1 Pro, Unspecified Version), viewport 1440x900) walking around the map for 30 s per mode, while 300 WebSocket bots walk around too. Entities / visible = avatars inside the server's area of interest / drawn on screen. Script: e2e/scenarios/perf.mjs; raw JSON in bench/results/browser-perf/.

This is an Apple M1 Pro with a fast integrated GPU, not the reference laptop (integrated GPU, 8 GB), and a single run per mode. It shows the renderer is far from saturated at this crowd size (economy mode is capped at 30 FPS by design); it does not show what a weaker laptop would do.

#Reading the numbers

• Bandwidth was the quadratic cost. 1,000 concentrated bots wrote 58 MB/s (about 465 Mbit/s) before and 11.5 MB/s (about 92 Mbit/s) after. On a hosting plan with an egress quota or a 100 Mbit/s port this is the number that matters, not CPU.
• Tick budget: at 15 Hz a tick has 66.7 ms. Before the change the worst tick of the 1,000 concentrated run (142.9 ms) exceeded it; after, the worst tick was 59.8 ms (mean 6.7 ms). That is inside the budget but with a thin margin on this laptop; a slower CPU would not have it.
• Latency did not move: median 30-46 ms, p95 about 64-67 ms, p99 about 68-82 ms on loopback. It is dominated by the 15 Hz flush (a state change waits for the next tick), so a network round trip adds on top. The 150 ms target has not been validated across a real network.
• Memory: server RSS 127 MB with 1,000 connected bots (Go heap about 34 MB at the end of the run); no server-side growth was concluded from these short runs.
• Server CPU (percent of one core, sampled with §ps§) was 45-54 % at 1,000 bots, but the generator was competing for the same cores, so this is an upper bound for the server and not a sizing figure.
• Mass reconnect (scenario E): 500 clients dropped at once and reconnected (with up to 2 s of client-side jitter, as the web app does) in 2.0 s, 0 failures, 0 kicked clients.
• Proximity grouping cost: 300 concentrated bots with media consent formed 64 groups; mean tick 2.15 ms, the same order as the same run without groups.

#Media through the SFU (scenarios B, C, D)

Synthetic but real media: bench/mediagen publishes an Opus track (~56 kbit/s on the wire) and a 640x360 VP8 track (~520 kbit/s) from encoded files, and every participant subscribes to the others through a local LiveKit server 1.13.7 and measures the RTP it actually receives (loss from sequence gaps, RFC 3550 jitter, stalls over 500 ms). Everything ran on the same M1 Pro laptop; the SFU CPU is percent of one core (100 = one core), sampled with ps, and the generator competes for the same CPU. Script: bench/run-media.sh.

• B = 5 calls of 4 (20 of the 100 present are in calls; the other 80 cost the app server only, see scenario A). C-10x4 and C-15x4 are scaled-down versions of scenario C (40 and 60 people instead of 100); C-25x4-audioonly is 100 people with audio only. D = one meeting of 20, each receiving at most 6 camera videos, one 720p screen share.
• Valid results are the ones with 0 % loss: B (20 people), C-10x4 (40 people) and D (20 people). The SFU used 46 %, 123 % and 152 % of a core while forwarding 35, 69 and 107 Mbit/s: roughly 1.3-1.8 % of one M1 Pro core per Mbit/s forwarded, in these three points.
• C-15x4 (60 people) already lost 34 % of the packets and the audio-only run with 100 people lost 61 % and could not connect 17 participants. With the generator on the same laptop we cannot tell how much of that is the SFU and how much the 100+ WebRTC clients (and their DTLS/SRTP work) fighting for the same cores and the loopback socket buffers. Scenario C at its full size (100 people in 25 calls of 4) was attempted and failed (first attempt with video: only 48 of 100 connected, 74 % loss; raw JSON kept in bench/results/media-invalid-full-c). It needs a generator on another machine.
• ESTIMATE, not measured: if one vCPU of the 2 vCPU reference server were 2-3x slower than an M1 Pro core (an assumption nobody has verified here), the same ratio would mean ~3-5 % of a vCPU per Mbit/s forwarded, i.e. roughly 40-65 Mbit/s (80-120 forwarded 500 kbit/s video streams) before the SFU alone uses both vCPUs. Treat this as a hypothesis to test on the real server.
• Not measured: TURN-relayed media, simulcast layer switching under congestion (the test tracks are single-layer, the browser client publishes two layers), a long run, packet loss injected on the network.

#10-minute soak (presence only)

300 bots for 600 s (300 joined, 0 failed, 0 disconnects, 0 kicked), server and generator on the same laptop, media disabled, sampled every 30 s (19 samples). Raw JSON: bench/results/soak/.

Tick p50 / p95 / max: 5 / 10 / 16.5 ms; propagation latency p50 / p99: 38 / 72 ms. Goroutines and RSS stayed flat and the heap oscillated around a stable level, so no leak showed up in ten minutes on this workload. Ten minutes cannot show a slow leak (hours) and this is not the two-hour soak.

#On a free cloud VM, over the internet

Measured on 2026-10-01 against the production Compose stack (Caddy with TLS, Tilework, LiveKit 1.13.7) on the public demo host: an Oracle Cloud Always Free VM.Standard.E2.1.Micro in São Paulo (2 vCPUs as seen by the guest, burstable, 1 GB RAM plus 2 GB swap). The generators ran on one laptop in São Paulo (Apple M1 Pro, home internet) and reached the VM over the public internet, so these numbers include TLS, the network and the shared, throttled CPU of a free VM. Every presence run used a fresh database (benchmark mode: deploy/docker-compose.bench.yml); media rooms were outside the office, so the app's reconciler left them alone. One run per row, so variance is unknown, and the 50 and 100 bot runs were taken about five hours before the others, so part of the jump in steal between 100 and 200 bots may come from the host's neighbours rather than the load. The laptop's own CPU and upload were not measured. Details and the image of each run: bench/results/vm-oracle-micro/README.md.

Host CPU is the share of all vCPUs busy; steal is time the hypervisor gave to other tenants, which on this burstable shape is consistent with its CPU cap. Docker's per-container CPU inflates under steal (it went past 100 % of a vCPU), so the table only gives the app's and Caddy's share of the CPU the two used together. Tick percentiles are not shown: the histogram's last finite bucket is 50 ms and the 200+ runs had a p99 above it. Scripts: bench/run-vm.sh, bench/run-vm-media.sh, bench/vm-sample.sh, bench/vm_report.py; raw files in bench/results/vm-oracle-micro/.

What it shows:

• Presence (no media). Every bot joined in every run and nobody was kicked. 100 bots was the last run where every tick percentile fit the histogram (p99 5 ms, max 82 ms, against a 66 ms tick period at 15 Hz). At 200 bots the mean tick was still 3.1 ms and the median move reached others in 88 ms, but p99 went above 50 ms, the worst tick took 180 ms and p95 latency rose to 230 ms. From 300 bots the busy share stopped growing at about 22 % while steal passed 70 %, consistent with the shape's CPU cap; at 500 the mean tick (99 ms) exceeded the tick period, so the server was saturated even though nothing failed. Caddy, terminating TLS for every WebSocket, used 36 to 57 % of the CPU the app and Caddy used together.
• Media. One meeting of 4 people with video and 20 people in audio-only groups of 4 ran with no loss that matters (0 % and 0.5 %). Twelve people with video lost 27 %, twenty lost 63 %, and the 60-person audio-only run (59 connected) lost 51 % at only 4.8 Mbit/s. Steal above 60 % in the worst runs makes the VM's CPU cap the most likely limit, but the generator laptop's CPU and home upload were not measured, so the generator side is not ruled out. These runs say nothing about a VM with dedicated CPUs.
• Reading it (estimate, not a capacity claim). On a free micro VM, about 100 people walking around stayed smooth and 200 still worked with slower worst cases; one small video call ran clean and three did not. Single runs, one region, one generator machine.

#Not run

#Reproduce

bench/environment.sh
OUT_DIR=bench/results/mine bench/run-presence.sh   # ~25 minutes
bench/compare.py baseline-json-15hz mine