What we withdrew

What we withdrew

This page lists figures we published, or came close to publishing, and then retracted — each with the reason. It is not an apology page. It is the part of the method that can be checked from outside: a benchmark you only publish when it flatters you is advertising, and the way to tell the difference is whether the retractions are visible too.

The short version. One measurement is the reason most of these fell: two byte-identical servers — same release, same configuration, same data — came out 11.6 % apart on 20 of 20 rows. Once a difference of that size can come from nothing but which physical host a machine landed on, most single-box comparisons stop being evidence.

The measurement that forced the rest

Two servers were rented as an identical pair — same release, same configuration, same data — specifically to find out what "identical" is worth as a baseline. They came out 11.6 % apart on the median, on 20 of 20 rows, in one direction.

Not the processor. Not contention from other tenants. Not memory fragmentation. Nothing we could attribute. Across three boxes the spread was about 19 %.

That result is the reason this page has seven entries. A one-directional difference across every row is exactly what identical servers produce, so consistency across rows is not evidence that the software differed — which is what several of the withdrawn claims had been resting on.

"+32 % from the distribution's own packages"

The claim was that taking every component from the operating system's own archive — rather than from a third-party repository or a compiled build — was itself worth about a third more speed.

It did not survive attribution. When the gap it came from was decomposed one variable at a time, the share that could be assigned to where the packages came from was 0 %: the PHP version accounted for nothing, and a PHP directory restriction accounted for about 60 % of the gap with the wrong JIT mode about 8 %. The remaining 32 % is published as unexplained rather than credited to the packaging. The aaPanel benchmark has that ladder in full.

The distribution-archive rule stays, because its actual benefit is not speed. It means a security fix arrives the way the rest of the operating system's fixes arrive, with no third-party repository in the serving path. That was always the reason; the speed number was never one.

"MariaDB 11.x is 2.7× slower"

This one was real, and it is narrowed rather than deleted. The 2.7× came from one report query on MariaDB 11.0's cost model. It was true of 11.0 and it says nothing about 11.8.

When 11.8 and 10.11 were put on identical servers with the same shop's 66,701 orders, the whole-application result was a wash.

The general point survives the number: newer is not automatically faster for a fixed application, because a planner change that helps most workloads can hurt one particular set of queries. What changed is that we measured again on 11.8 and the effect was not there. The version pin it justified is retired.

The database comparison that was thrown away entirely

This is not a number that was corrected. It is a comparison whose rows were discarded, because the harness was timing an encryption handshake and calling it database speed.

The tell was in the control. The battery's floor — SELECT 1, a statement that does no work at all — read 12 ms on one arm against 29 ms and 37 ms on the other two. A floor that moves three-fold is not query execution. Nothing about running SELECT 1 should differ between two database versions by that much, so the measurement was not measuring what its labels said.

The cause: MariaDB 11.8 negotiates TLS on a unix-socket connection where 10.11 does not. Neither box asks for it — 11.8 auto-generates a certificate and enables it, and nothing on either box requires encrypted transport. It happens because both ends support it. Over the socket, one arm reports no cipher at all and the other reports TLS_AES_256_GCM_SHA384.

Held constant: same statement, same dataset, 30 invocations of the client per arm.

Connect cost alone is 8 ms on 10.11 against 22–25 ms on 11.8, and 6 ms with TLS off.

The consequence is the honest headline: every raw database row taken before that fix had to be re-measured. So there is no corrected per-version total to publish here, and none is offered — the rows were thrown away rather than adjusted.

Your shop was never affected. PHP's database driver does not negotiate TLS over a unix socket — measured through the real driver on both boxes, the connection reports no cipher and costs 0.12–0.21 ms. What did pay it was the panel, which reaches the database by invoking the client program rather than through a driver. Fixed by passing the TLS-off flag at every local call site: a unix socket on the same machine, scoped by file permissions, where the encryption protects nothing.

"PHP 8.5 cannot run 6amMart"

Disproved, and it had been on this site.

The reason it was believed was specific and plausible. The spreadsheet library behind Bulk Import, Bulk Export and every Excel report declares that it needs PHP below 8.5. A failure there is the worst kind, too: the APIs and the storefront keep looking healthy while nobody can download a report.

But a declared bound in a library's metadata is not a measurement, and nothing enforces it at runtime because the platform check is switched off. So the code was run on PHP 8.5.4 instead of read. The xlsx round-trip came back byte-identical, including Devanagari text and an em dash, and the framework boots. Checked on both the plain CodeCanyon tree and the optimised fork, because a result on our fork would have proved nothing about what most buyers run.

Ubuntu 26.04's archive PHP is 8.5.4, so this was not academic. While the claim stood, the recommended operating system was one the software would not install on.

"PHP 8.4 is about 8 % faster than 8.3"

Withdrawn, because it was an artefact of the measurement order rather than of PHP.

Interleaving the two arms is not sufficient on its own. A fixed A-then-B order within each pair puts every warm-up on the same arm, which hands that arm a systematic advantage on every pair. Re-run with the order alternating within the pair as well as between pairs, the result was a wash.

The 8.3-against-8.4 figure that stands was taken on one machine that had both installed, with the 8.3 pool given the entire 8.4 tuning first and every setting verified from the running process: geometric mean 0.9994, with 8.3 ahead on 3 rows of 7 and every difference smaller than at least one arm's own spread.

"This kernel is about 10 ms slower per request"

Withdrawn. It was never published as a claim — it was recorded as an observation we declined to claim, and the consistency was the only reason it seemed worth mentioning: about 10 ms slower on idle serial latency, on every run, on two different endpoints, with non-overlapping ranges.

Then the identical-box pair came back 11.6 % apart on 20 of 20 rows. With one machine per operating system, nothing separated "this kernel is slower" from "this host was slower", and the identical-box result says the second explanation is entirely capable of doing all the work.

Recorded as retired rather than deleted, because an earlier version of the operating systems page mentioned it.

Storage speed, from all of those rounds

Unusable, not corrected. Those figures reflect which physical host each virtual machine happened to land on. There is no version of the measurement that recovers a software conclusion from them, so none is offered.

Six axes swapped, nothing reportable on five

This is the fuller context for the entries above. The operating system, the PHP version, the database version, the web server, the cache and the kernel were each swapped and measured on this application.

Exactly one of the six decided anything — the cache, and on compatibility rather than speed. Nothing else produced a difference worth reporting.

Held constant: the same 6amMart, the same real dataset, identical 4 GB / 2-core servers, 20 concurrent users on the search endpoint for 30 seconds.

The spread across all three is 3.5 %, which looks like a result until the next number: Ubuntu 26.04 disagreed with itself by 8.6 % across two runs of its own identical configuration — 49.6 req/s once, 53.9 the next time.

So the correct reading of that table is "about 53 requests per second on 2 cores", and nothing more. It ranks nothing.

Two things withdrawn before they shipped, not after

Not every retraction is of a published number. Two hardening proposals were rejected by measurement rather than shipped: hiding the web server's configuration closed nothing and broke the readiness tool, which reads it; and blocking the panel's own directory wholesale would have taken the database administration tool down with it.

One database index was rejected the same way. A candidate index for two catalogue listings measured 15 ms down to 12 ms unforced, 16 ms when forced, with examined rows unchanged — the query is rewritten to drive from a different table, so the plan the index existed to enable is unreachable. An index that does not change how many rows are examined has not changed the work.

And one performance figure was corrected against a tool of our own rather than a server: a readiness probe reported a report page at 468 ms for a page that runs at about 340 ms, because the probe used tidy inner joins where the application writes outer ones. The application's own join shape measures 335 ms. The probe was wrong, not the server.

One that was not a measurement at all

The sizing rules a server is configured from existed in two implementations — one in the installer, one in the panel — each carrying a comment telling the reader to keep it in step with the other. They were not in step. Run against each other across 36 box shapes, 66 of 432 values disagreed, in nine distinct categories: one worker calculation was 3× apart on a six-core box with three shops, and one table-cache figure differed on 27 of the 36 shapes, in both directions.

Nothing was withdrawn here because nothing had been claimed. It is on this page because it is the same failure as a bad benchmark — two things believed to agree, never actually run against each other. There is now one definition, and a test that executes both real call sites across all 36 shapes and requires every value to match.

Related

How we measure is the method these were checked against · The aaPanel benchmark is where the attribution ladder lives · Operating systems · Database versions · PHP versions.