How we measure

How we measure

Every figure in this section came from a round that was run on real servers, against a real shop's data, and written up before anything was published. This page is the method: how a comparison is arranged, which numbers are allowed to count, and the specific traps that produced wrong answers before the arrangement was fixed.

It is here because the arrangement — not the numbers — is what decided most of these rounds. A fixed pass order manufactured an 8 % PHP win that vanished when the order alternated. A restored set of database files manufactured a 5 % version win. An error page answering in 0.3 ms manufactured a spectacular database result. None of those were faults in the measuring tool; all three were faults in how the comparison was set up.

The rules, in short

The noise floor, and why it dictates everything else

This is the single most consequential number in this whole section, and it is about the measurement rather than the software.

Two servers from the same provider, with the same processor model, the same clock speed, the same kernel, the same release, the same database build, the same buffer pool, the same worker count, the same cache settings and the same data, measured in the same wall-clock window, came out 11.6 % apart — on every row, in the same direction.

That was checked for the obvious causes and none of them held. A benchmark touching neither PHP nor the database matched within 1 % across all three machines, so it is not processor speed. Hypervisor steal time read 0.00 % and 0.02 % at idle, so it is not a noisy neighbour. The database files were identically fragmented. Nothing we could measure accounted for it.

So on separate machines, an effect smaller than about 19 % cannot be attributed to software. That is why almost every axis in these rounds was measured by swapping a component inside one machine, and why the one axis that could not be — the operating system itself — is reported as a direction and never as a number.

Control rows are the resolution floor

Every comparison includes rows that execute no PHP and touch no database: a static file served straight by the web server, and micro-cache hits that never reach PHP at all.

Those rows should not move when a PHP version, a database version or a kernel changes. Whatever they do move by is that arrangement's floor.

Three things they tell you, each of which changed a conclusion in these rounds:

An effect that does not clear its own control rows is not reportable. That single reading turned the web-server and kernel axes from "small win" into "wash".

Control rows moving with the treatment means the window drifted, not the treatment. That is how the fixed-order PHP error was caught: rows where no PHP runs moved 10 to 13 % in the same direction as the PHP rows.

Control rows moving further than the effect means the signal is below the floor. On one web-server comparison the controls shifted +7.2 % — seven times the app-row difference, in the opposite direction.

One caveat we got wrong first and correct here: the storefront home page is not a PHP-free control. Its server-side render fetches the admin API, so it executes PHP indirectly. Only the static file and the cache hits are genuinely PHP-free, and they are the smallest numbers on any table — which is its own hazard. At 0.4 to 1.3 ms, one tick of timer granularity is 13 %, so a sub-millisecond row is never read as version signal.

Order matters more than repetition

Alternating passes between two arms looks balanced. It is not, if the order inside each pair is fixed.

Everything that warms up during a pair — database buffer pages, the opcode cache, the kernel's page cache for the storefront's asset chunks — then lands on the same arm every single time. That is not a small effect:

The correct arrangement gives each arm first position half the time: A then B, B then A, A then B, and so on. Whichever arm runs second in a pass looks better, and that is a warm-up component sitting on top of whatever the real difference is.

For a three-way comparison the same principle applies: the starting series rotates every round and reverses on alternate rounds, so each arm takes each position an equal number of times.

When a live swap is impossible: A, B, A

Some things cannot be swapped inside one boot or inside one pass — a kernel needs a reboot, a database version swap takes minutes and rewrites files. Alternating is unaffordable there.

So those run A, B, A: the original, the new one, then back to the original. The third leg is the control, and it is not a formality.

It caught a 5 % version "win" that was drift. On one database comparison the new version came out 2.7 % ahead on 14 of 14 rows, which looks unambiguous. Then the original version, measured again, came out 5.0 % faster than itself — a bigger gap than the one separating the versions — and beat the new version on 10 of 14 rows on its second outing. Twelve of 14 rows got monotonically faster across all three legs regardless of which version was running.

And it caught a kernel result that was the machine drifting. The shipped kernel, measured again at the end, came back 6.7 % slower than the number it was meant to reproduce — four times the effect being measured — with its control rows 14.5 % adrift and winning zero of 14 rows. If the kernel had been responsible, the third leg would have returned to the first.

The freshness trap that makes the third leg mandatory

A freshly restored set of database files is about 8 % faster than one that has been in place. A logical restore rewrites every table and index compactly: no fragmentation, no leftover page splits, statistics fresh.

That is large enough to be a whole result on its own, and it is how a "12.3 is 5 % faster than 11.8" reading was produced. The same 11.8, re-measured back-to-back on freshly restored files, came out 7.8 % faster than itself — monotonically, across nearly every row.

So: any comparison where one side has been dumped and restored and the other has not is comparing file freshness, not versions. If a version move requires a restore in one direction, restore both sides before timing them, or compare only moves of the same shape. And re-measure the first version at the end, or "freshness" and "version" are the same number.

Every row carries its status codes

An error answers very fast. Measured: a server error in 0.3 to 0.9 ms, against a working storefront's 29 ms. So a row whose requests were failing does not look broken in a results table. It looks spectacular.

This is not hypothetical. It happened three times in these rounds:

• Six passes of one database leg recorded 30 of 30 requests failing at a p50 of 0.3 ms on the storefront row, because an earlier failed swap had stopped the writers and never restarted them. That row was making its whole leg look faster. Recomputed on the rows that were clean successes in all three legs — still a wash, but the earlier printing of that table is withdrawn.
• One equalised operating-system run had the storefront stopped entirely, by a database upgrade that quiesced it and did not bring it back. Restarted and re-run.
• One comparison had a build running on the machine that was serving a measurement pass. Its storefront row read 0.3 ms and its concurrency-4 rows were 40 to 50 % slower from the build's load.

So a row whose status codes are not entirely successful requests is dropped, not averaged, and the gate that starts each leg now requires the admin panel and the storefront both answering, the exact expected row counts, and every service active before the first pass is written.

A related trap, worth naming because it looks identical to an outage: a wrong hostname in one gate produced a connection with no response at all. On a correctly configured server an unmatched hostname makes the web server close the connection, which is indistinguishable from nothing listening. It reported "the application is down" on a machine answering normally. Gates now verify a hostname against the server's own configuration before dialling it.

Prove the arm, do not trust the swap

Three consecutive component swaps once silently did not happen, and the comparison measured one arm twice under two different names — reporting a wash as a result.

The cause was a shell pipeline: a version-reading helper closed its own pipe early, the writer took a signal, the function returned a failure code, and the calling script sent that output to a null device and trusted the exit status.

Now every pass reads the version and the relevant build flag back off the running binary into a sidecar file, and a pass whose arm reports the same version as the other arm is skipped and named. An arm with no proof is not evidence, and one earlier result was thrown away and redone for exactly that reason.

The same principle applies upward: a package's metadata is not the artifact. One runtime's own release was published with a version string reading -alpha inside an otherwise normal build, and nothing in the package metadata showed it — only running the binary and asking its version did. Ask the binary, never the package version.

Two rate limiters, one fake speedup

Worth knowing before running any load test against this application: the framework throttles at 600 requests per minute on top of whatever the web server does.

A naive local load test therefore measures the throttle, not the application, and a change that alters throttle behaviour reports as a speedup that no customer would ever see. Requests that were throttled are counted separately and never averaged into a result, a distinct authorisation token is used per request where the throttle keys on it, and load tests that matter are measured through the real edge rather than over the loopback interface.

There is a related hazard in the other direction. The web server may cache some responses even when the request carried an authorisation header, so a row that exists to force execution through to PHP can silently be answered from cache instead. One run printed 0.7 ms and 31.8 ms for two rows it labelled identically — the first was 12 cache hits. Fixed by adding a unique argument that is part of the cache key and read by no handler, verified by requiring the response body to stay byte-identical with and without it, plus a guard that makes any such row report its own invalidity rather than a number. That was a 31× correction on one row.

What is held constant

Every arm of every comparison in this section ran with the same tuning. Not re-derived per arm — copied, then verified by reading it back from the process that serves requests.

An arm that fails that check is fixed and re-measured, not reported. Mirroring is not optional: a freshly installed PHP pool has none of that tuning and a lower worker cap from its default configuration, so timing it against a tuned pool measures the tuning.

One reason this matters more than it looks: rebuilding a configuration cache changes nothing at all until the PHP process pool reloads, because timestamp validation is off. A setting can be correct on disk and absent from the running process.

The dataset

One real production dataset, staged to every machine from a single dump so the bytes are identical, and asserted by row count before any arm counted:

| | | |---|---| | Orders | 66,701 | | Order lines | 113,231 | | Customers | 17,534 | | Tables | 192 | | Database size | ~415 MB | | Uploaded files | ~901 MB |

Hardware, on every machine in these rounds: 2 virtual cores, 3.9 GB RAM, same provider, same processor model, same region, non-rotational disk. Verified rather than assumed — same model, all pinned at the same clock, all the same disk size and type.

We have not measured 8 GB or larger servers, so nothing in this section predicts them.

What is measured, and what it is not

The figures are time to first byte, at the median and the 95th percentile, with a warm-up request per URL discarded, over ten URLs: five public API endpoints, an authenticated order history, the admin entry page, the server-rendered storefront home, a static asset served by the web server, and a websocket handshake. Each is measured at concurrency 1 and concurrency 4.

Cacheable URLs are measured twice — once anonymously, where the cache may answer, and once forced through to PHP. Without that split the cache hides every application-level difference.

TLS handshakes are measured as their own axis rather than folded into the application rows, because the releases ship different TLS libraries and folding it in would smear one cause across ten numbers. That axis came back inconclusive at this sample size and is reported as such: the two releases looked 15.7 % apart, until the two identical machines in that same test differed by 13.3 %, and the same release comparison read +15.7 % against one reference machine and +2.1 % against the other.

Every claim about the application is checked in its own source

A benchmark cannot tell you that an application will not run. Two of the most consequential findings in these rounds came from reading the code, not from timing it:

MySQL was ruled out by finding the exact line where the application switches off a MariaDB-only optimiser setting — which MySQL rejects outright, making the shop's search endpoint fail on every request — and the four migrations that read the server's own catalogue in a casing MySQL disagrees with. Neither is guarded, and both are silent on MariaDB, so nothing in a normal build surfaces them.

PHP 8.5 was cleared the same way in reverse: a library's declared version bound had been read as ruling it out, so the code was run instead of read, and the output compared byte-for-byte on both the plain vendor tree and the optimised one.

The raw data is archived, and the published tables are re-derivable

The servers that produced these numbers were deleted when the round closed. Immediately before that, the evidence was archived: 269 files — the raw per-pass results, the swap transcripts, the proof-of-arm sidecars, and the measurement harness itself.

It carries a short guide showing how to recompute a published table from the raw files, and that recipe was verified to reproduce one of the published PHP tables exactly.

That matters more than it sounds. Without it, these reports could be read but never re-derived, and a number nobody can recompute is a number nobody can correct. Several of the corrections on these pages exist because someone went back to the raw files and found a row whose requests had been failing.

The archive also carries the two exclusions honestly. One set of web-server passes is deliberately left out of the published result, because it predates the proof-of-arm sidecar and so cannot show which binary served it. And the machines used for these comparisons carried extra package sources left by the swap harnesses, which is stated so nobody mistakes a test machine's package list for what a real install produces.

What we do not publish

A number we cannot source. If a figure is not in a round's written record, it does not appear on these pages, and we do not average two figures into a third or convert a percentage into a multiple to make it sound larger.

A magnitude the arrangement cannot resolve. Where only the direction survives, only the direction is stated, and the noise floor is stated in the same breath.

A result whose control failed. Those are reported as washes, which is what they are.

A claim quietly dropped. Where an earlier statement turned out to be wrong, the correction is on the page along with what produced the error, because the wrong reading was the easy one to reach in most of these cases and someone else will reach it too.

Related

The OS and stack round is the round these rules were mostly written during. MariaDB versions, measured and PHP versions, measured are the two axes where the arrangement changed the answer.

For the decisions rather than the evidence: which operating system and which database engine.