Reports

Long-running capacity scaling tests.

The committee was grown from a single node to ten across three continents under saturating load, a relay and two edge leaves were attached, and then it was torn back down to one — a full 1→10→1 ladder. Twice, independently, on real cross-region infrastructure.

Each stage had to clear three gates before the ladder moved on: a committed-TPS floor, a committed-bytes floor, and chainHash agreement across every node. Both runs cleared all of them. The second run exists because one clean pass does not distinguish a fix from luck.

These runs are complete and final — the fleet was destroyed after the second report, so the numbers below will not change. Both ran the same build (9a5d58c) against the same genesis configuration.

Reading the tables

TPS is committed transactions per second, taken from each node's [block-commit] lines rather than from client receipts — what the chain agreed, not what was offered to it. n is the committee size. p50/p90 are submit→commit latency in milliseconds over a closed-loop cohort; a dash or n/a is the known small-cohort measurement artifact at large committee sizes, not a missing gate — the gates were still evaluated.

The runs

Run 1 · 22 July 2026 · automated harness PASS all 22 stages green

Result: Pass
Duration: 2 h 07 m
Fleet: 13 nodes
Committee: 1 → 10 → 1
Regions: 3

17:49:35Z → 19:57:15Z UTC. This was the acceptance retest after three consensus and persistence fixes landed that day. Its predecessor, on the same fleet, had failed fatally at stage 07 on a leader-election divergence wedge; this run completed the entire ladder.

Fleet

Thirteen c5.xlarge instances (4 vCPU, 7.5 GB), 48 GB EBS each, Ubuntu 24.04 on OpenJDK 21: ten committee nodes q0–q9, one relay r0, and two leaves l0/l1 that dial only the relay. Committee members were spread round-robin across eu-central-1, us-east-1 and ap-southeast-1.

This is an inter-continental deployment, not a LAN rig. WAN round-trips dominate quorum-certificate formation, which is why committed throughput here is round-trip-bound rather than CPU-bound.

Load

Open-loop TpsClient with 256 in-flight requests per client, a 90-second flood per committee checkpoint, and 3 to 6 clients scaled by committee size so offered load stayed above single-node capacity. Mixed workload — core owner claims plus application-ledger transactions with id-only bodies — submitted round-robin across every committee endpoint.

Per-stage results

Stage 11 attaches the relay and leaves; FINAL re-verifies convergence.
Stage	n	Regions	TPS	Committed B/s	p50 ms	p90 ms	Gate	Change
01	1	eu	923.0	193 854	706	806	PASS	baseline
02	2	eu,us	354.2	74 385	812	856	PASS	+q1
03	3	ap,eu,us	547.3	114 932	1021	1037	PASS	+q2
04	4	ap,eu,us	558.9	117 379	997	1229	PASS	+q3
05	5	ap,eu,us	620.6	130 342	1018	1846	PASS	+q4
06	6	ap,eu,us	457.8	96 140	—	—	PASS	+q5
07	7	ap,eu,us	504.0	105 853	—	—	PASS	+q6 — prior run's FATAL point; clean here
08	8	ap,eu,us	471.8	99 090	1069	1212	PASS	+q7
09	9	ap,eu,us	675.2	141 794	n/a	n/a	PASS	+q8
10	10	ap,eu,us	541.2	113 659	—	—	PASS	+q9 (full committee)
11	10	ap,eu,us	781.2	164 060	1224	3031	PASS	relay + 2 leaves attached; convergence PASS
12	9	ap,eu,us	777.1	163 204	1223	1330	PASS	−q9
13	8	ap,eu,us	713.0	149 742	1160	1313	PASS	−q8
14	7	ap,eu,us	631.1	132 551	1140	1227	PASS	−q7
15	6	ap,eu,us	550.8	115 677	1112	1156	PASS	−q6
16	5	ap,eu,us	373.8	78 526	931	1016	PASS	−q5
17	4	ap,eu,us	284.9	59 857	999	1119	PASS	−q4
18	3	ap,eu,us	284.2	59 695	1027	1123	PASS	−q3
19	2	eu,us	197.6	41 533	819	920	PASS	−q2
20	1	eu	1000.3	210 094	714	756	PASS	−q1 (single node)
FINAL	1	eu	1009.8	215 118	679	760	PASS	1 quorum + 2 leaves via relay; convergence PASS

Single-node throughput sits at roughly 920–1010 TPS, falls towards a WAN-bound floor as the committee grows, and recovers on the shrink path. Every stage passed all three gates.

Overload behaviour

In the multi-node WAN stages offered load exceeded committed throughput by roughly 2×, and the bounded mempool applied 503 queue_full backpressure — 143 events across the run — while the chain kept committing. Committed TPS is therefore a saturation ceiling, bound by BLS verification, block apply and WAN round-trips, rather than a limit on offered load.

What this does not prove. The offered-load figures are approximate: the client double-prints its summary, and committed TPS is measured over the block-commit span rather than the full 90-second wall window. At n=1 the relationship is measurement-sensitive rather than a clean overload demonstration. The robust evidence is the committee stages plus the presence of backpressure. Turning “holds under higher load” into a measurement needs a dedicated fixed-committee overload-curve stage, which this harness does not have.

Correctness

Across all ten committee nodes: zero pin-rejects, zero whole-block rejects, and zero stall-watchdog trips. Each node fired 3 to 13 explicit ownership checkpoints with no watchdog, deadline or hard-cap trip, and [qc-signers] was logged on every committed block — the parent-QC signer bitmask round-tripping correctly on every cross-node proposal, with no duelling proposers. Stage 07, the prior run's failure point, committed cleanly.

Storage

Every node ended below 12 % of its 48 GB volume. On q0, the busiest committee node, the chain database held 668 MB of backing store against 0.28 GB of logical committed payload — HSQLDB row and index overhead, not payload growth — across 8 703 blocks and 1 418 499 transactions. The redo log stayed at about 88 MB against a 512 MB threshold, sawtoothing rather than growing: the unbounded-growth failure mode the checkpoint discipline exists to prevent did not occur.

Verdict

Acceptance pass. The full ladder plus relay and edge topology completed with every gate green, at throughput consistent with the established baseline, with all three of that day's fixes validated on live cross-region infrastructure.

Run 2 · 22 July 2026 · operator hand-run PASS all 22 stages green · independent reproduction

Result: Pass
Duration: 2 h 09 m
Fleet: 13 nodes
Committee: 1 → 10 → 1
Regions: 3

20:10:58Z → 22:19:52Z UTC. A second, independent acceptance run, launched by hand on the same fleet directly after the automated pass, with the default harness invocation verified from the live process — no dry-run, no stage skipping, no environment overrides. Same fleet shape, same load profile, same build.

Per-stage results

Stage 11 attaches the relay and leaves; FINAL re-verifies convergence.
Stage	n	Regions	TPS	Committed B/s	p50 ms	p90 ms	Gate	Change
01	1	eu	924.6	194 187	678	696	PASS	baseline
02	2	eu,us	301.2	63 259	818	853	PASS	+q1
03	3	ap,eu,us	490.3	102 971	1022	1064	PASS	+q2
04	4	ap,eu,us	569.1	119 519	1022	1036	PASS	+q3
05	5	ap,eu,us	438.8	92 157	—	—	PASS	+q4
06	6	ap,eu,us	401.7	84 375	—	—	PASS	+q5
07	7	ap,eu,us	536.8	112 742	—	—	PASS	+q6 — prior-run FATAL point; clean
08	8	ap,eu,us	724.4	152 133	—	—	PASS	+q7
09	9	ap,eu,us	650.4	136 581	n/a	n/a	PASS	+q8
10	10	ap,eu,us	658.9	138 384	—	—	PASS	+q9 (full committee)
11	10	ap,eu,us	773.5	162 443	1227	1328	PASS	relay + 2 leaves; convergence PASS
12	9	ap,eu,us	788.2	165 545	1228	1283	PASS	−q9
13	8	ap,eu,us	750.4	157 587	1221	1338	PASS	−q8
14	7	ap,eu,us	731.7	153 670	1131	1229	PASS	−q7
15	6	ap,eu,us	568.3	119 357	1124	1228	PASS	−q6
16	5	ap,eu,us	426.5	89 573	1025	1102	PASS	−q5
17	4	ap,eu,us	362.8	76 201	1047	1126	PASS	−q4
18	3	ap,eu,us	291.9	61 329	1024	1055	PASS	−q3
19	2	eu,us	201.7	42 382	845	920	PASS	−q2
20	1	eu	999.1	209 839	715	750	PASS	−q1
FINAL	1	eu	999.0	212 818	715	738	PASS	1 quorum + 2 leaves via relay; convergence PASS

The profile tracks run 1 closely — n=1 at 924 rising to 999 on the way back down, n=10 at 659, the relay stage at 773 — which is the point of the run: cross-run stability, not a new number.

Correctness

Sampled across q0, q5 and q9: zero pin-rejects, zero whole-block rejects, zero stall-watchdog trips, [qc-signers] on every committed block, and explicit checkpoints firing with no trips. Stage 07 clean again.

Storage

All nodes again ended below 12 % of 48 GB. q0 committed 8 231 blocks and 1 470 096 transactions for 0.29 GB of logical payload against 710 MB of backing store, with the redo log bounded at 174 MB under the same 512 MB threshold.

Verdict

Acceptance pass, second independent run. Two consecutive clean end-to-end passes on real cross-region infrastructure. The fleet was destroyed after this report.

Provenance

Both runs came from the 130-ec2-scaling-test harness (run-scaling-test.sh). The numbers above are transcribed from each run's report.md, which is kept alongside this site's source together with the run manifests. The raw evidence — per-node DEBUG logs and per-client submit summaries, several gigabytes per run — is archived outside the repository.