Security and isolation, measured

Security and isolation, measured

Every project on a SixPanel server gets its own unix user, its own PHP-FPM pool, and its own password-protected Redis instance with kernel-sandboxed background workers. This page is the evidence for that design and the audit that produced it — including the cost, because hardening is not free and publishing only the benefit would be advertising.

The three answers worth having up front. Every cross-project read attempt failed, and what refuses it is the kernel rather than a check inside PHP. The whole security layer costs about 20 microseconds against a 26-millisecond page — roughly 0.08 % — measured as CPU time per request rather than throughput. And a six-dimension audit raised 44 findings, of which 38 survived an independent skeptic whose default was to refute them.

What was actually attempted, not asserted

"Isolated" is a claim that can be tested, so it was. Every read below was attempted from inside one project's PHP, on the SixPanel box and on a tuned aaPanel box of the same specification, with the same shop's data.

Every cross-project read failed on both machines. What differs is the kind of refusal, and that difference decides what happens when something goes wrong.

On aaPanel every site runs as the same unix user with world-readable files, so a PHP-level directory restriction is the only barrier — a check inside the process that is being asked to respect it. On SixPanel the refusal comes from the kernel declining a different user id.

Where the two overlap they part company on one measured case: a 0600 file another site had written into the shared temporary directory was readable on aaPanel and denied on SixPanel.

The permission modes behind that, as installed:

The leak side and the attack side are different questions

This is the finding that changed how the audit was run, and it is worth more than any single number on the page.

Reading the isolation result as "nothing leaks, therefore isolated" was wrong, because it only asked one of two questions. What a project's PHP could not read was every secret: the panel's own settings, the system password file's shadow, every other project's settings file. What it could read was the machine — the web server's configuration, every site definition the panel writes, the other pools' user names and socket paths, the panel's own code, the shared temporary directory and its other users' files, and every other process's command line.

None of that is a secret. All of it is reconnaissance handed to whoever gets code execution through an uploaded theme or a plugin. And one path mattered more than the rest: the directory holding one site definition per project names every domain on the server.

The same distinction cut the other way on the vendor codebase, which is why it is a method point rather than a one-off. An open image proxy in the untouched CodeCanyon storefront was rated high rather than critical for a precise reason: exploiting it costs the owner processor time, disk and reputation — arbitrary remote content re-served under their own certificate — but not data. And a separate gap where any .php file under the web root would execute was rated the same way: a defence-in-depth gap that needs a prior arbitrary-write foothold, not a live remote-code-execution hole.

What the kernel namespace closed, and what it cost

The reconnaissance surface above was closed with a systemd mount namespace on the worker pool. The cost was measured before the change was accepted, on the same box and the same requests.

Held constant: same box, same shop, same endpoints, before and after the drop-in.

The last two rows are the point. The confinement is drawn once, when the process starts, so it does not appear in a per-request measurement at all. The same treatment was later applied to the four per-project background workers: the panel's own directories became unreadable to them, visible processes went from 141 to 5, and the application tree stayed writable.

Two directives were deliberately left out, each for a measured reason. One that would have blocked writable-executable memory is excluded because it disables the JIT mode that measures 6.0 % on this application's PHP endpoints — that is a real trade, and it was made in favour of the speed. Two paths cannot be closed at all: the system password file, because the C library needs it to turn a user id into a name, and the pool directory, because the worker master needs to know which pools to run. Both are root-owned and read-only.

Two further proposals were rejected by measurement rather than shipped: hiding the web server's configuration closed nothing and broke the readiness tool, which reads it; and blocking the panel's directory wholesale would have taken the database administration tool down with it, since that runs as another pool of the same service.

Per-project Redis, and the ACL that does not exist

This is the clearest case on the page of a design forced by a measurement rather than chosen.

The original arrangement was one Redis instance behind one password, and every project's own settings file had to contain that password for its own application to work. Which means the password isolates nothing: starting from one project's unix user, a probe read another project's cached SMTP password, Firebase credentials, Maps keys, SMS gateway key, reCAPTCHA secret and two payment gateways' secrets — and writes were permitted, so the victim's cached mail configuration was poisonable.

The obvious fix is an access-control list scoping each project to its own database number. It does not exist. Redis 7.0 — which is what Ubuntu 24.04's archive ships, and the archive is where every component comes from — has no per-database ACL. A key pattern applies in every database, and the commands that enumerate keys leak key names whatever the patterns say. Denying the one command that would stop one shop emptying another's cache breaks the admin panel's own "clear cache" button, on both the optimised and the untouched codebase.

So the boundary is the filesystem, not the password. One instance per project, its socket inside a runtime directory group-owned by that project alone. A process running as another project's user cannot traverse that directory, so it never reaches the protocol — knowing the password would not help it. The per-instance password is the second lock.

The reproducer went from failing to passing in both directions: refused with a permission error while holding the target's own password.

What it costs, measured on a 2-core box:

The change also retired a 15-project ceiling and the stack-wide shared password. And moving to a unix socket was faster rather than slower, though not in a way a shopper feels: at 50 concurrent clients, SET 40,053 to 66,519 operations per second and GET 48,780 to 65,789, with single-client GET median 0.039 ms to 0.031 ms. At the endpoint the same change measured 19.1 to 19.6 ms on one arm and 54.4 to 56.2 ms on the other — both inside run-to-run spread, so no endpoint claim is made from it.

What the hardening costs

The whole security layer was measured, because a claim that it is free needs a number behind it.

The first attempt was thrown away and is worth saying so. It measured throughput, and on a 2-core box the benchmark client competes with the web server and PHP for the same processors: turning security off made the box slower in 5 of 7 per-control samples, which is impossible. Those numbers are not reported.

The reported run measures processor time per request, read from the service manager around an exact request count, so the client's own work cannot enter it and contention cannot fake a win. Every state is interleaved, and the shop is verified answering normally after every toggle.

Held constant: Ubuntu 24.04, 2 cores, 3,915 MB, a real shop with 190 tables, client running on the box itself over loopback with the shop's real hostname — so no network is in the figure. One /login/admin page: a genuine PHP request with a session and a rendered view.

Four interleaved pairs, in microseconds per request:

        web server                        PHP workers
ON      384.6  390.8  381.6  345.4        25155  26784  25489  25007
OFF     364.6  363.9  323.1  371.0        25753  26576  25572  25966

PHP is higher with security off, which is the honest way of saying there is no PHP-side cost to measure. The entire difference is on the web server's side. A static file, which never reaches PHP, moved 60.5 µs to 58.6 µs — about 2 µs.

Per control, on the PHP path, two samples each:

Those per-control figures sit at the noise floor and add up to more than the 20 µs the aggregate showed, so they are ceilings rather than a budget. The true total is under about 100 µs — under 0.4 % of a page.

Why the PHP side is free

Because there is no open_basedir. That setting validates a path on every file operation, and it is the single largest measured item in a competing panel's slowness: about 60 % of a gap that had been sitting unattributed, worth 17 to 23 ms per request with filesystem lookups up 19- to 33-fold. The aaPanel benchmark has that measurement in full.

The same boundary is drawn once, in the kernel, by the sandbox above — and the table two sections up is what "once" costs: nothing per request. That single choice is why the PHP column shows no cost to find.

The limiter gives CPU back

Loopback is exempt from every rate limiter by design, so the run above measured the lookup and never the accounting. With the exemption lifted for the length of one test and 4 concurrent clients pushed at a 120-per-second zone:

600 requests -> 200: 225   429: 375     (3,370 per second offered)

62.5 % refused, and a refusal costs about what serving a small static file costs. Under attack this control saves processor time rather than spending it.

Not in that number

TLS is in both baselines, so it is not attributable to hardening — and it is not optional. The CDN is bypassed by a loopback client entirely; measured separately from a real connection, an edge cache hit answered in 341–368 ms against 456–595 ms when it had to reach the server, so roughly 150–200 ms saved on requests that never arrive at all. Two-factor authentication, panel bytecode and signed manifests are panel-only and never on a shopper's path. And everything above was measured at an idle load average — under real concurrency the relative share should hold, since all of it is fixed work per request, but that is reasoning rather than a measurement.

The audit that produced this

Six dimensions: kernel and firewall posture, rate limiting, anti-tampering, application and server protection, setup-flow completeness, and code health. Every finding was then re-read by an independent skeptic whose default was to refute it.

The most serious of them is a good illustration of what an audit is for. Every rate limit tier — authentication, search, writes — was bypassable by prefixing the request path with the front controller's own filename. The limits were all correctly configured and all of them could be walked around.

The refutations are worth as much as the confirmations, and two are worth naming. A proposed host firewall turned out to have a default rule set byte-for-byte identical to the box's existing reachability — it would have added a component and changed nothing. And a proposed rate limit on the image endpoint would have cut a real shopper's image fan-out 4× on the first page they loaded.

The defect that mattered most was not a security finding

A project's installation ran against a different project's database. Measured on a project whose own settings file correctly names its own database:

The cause is a genuinely subtle one: the panel's service unit loads the panel's own environment, and the framework loads its settings file in immutable mode — a name already present in the process environment is never overwritten by the file. The same mechanism, found earlier, had a security consequence too: per-project background workers were running against the first project's database as the first project's database user, and were additionally handed the database root password and the panel's own admin password. Six colliding names plus two panel secrets. The default project was shielded by accident, because its cached configuration means the environment is never consulted — and the untouched vendor tree, which must not be config-cached, was fully exposed.

That is a standing reason this product is tested on both codebases rather than one.

Two related classes worth publishing

A file mode that reported green. A project's settings file ended up owned by root at 0600 while PHP runs as the project user, so the application could not read it. The shop worked for ninety minutes on a cached configuration file that root had written 0644 with the database password inside it; the first update cleared that cache and every page failed, naming the framework's built-in default database — which is the proof that the settings file was never read at all. The panel's own health check read the file successfully every time and reported green, because the panel runs as root. A root-owned tool cannot answer "can this application read this file." The check now asks the application.

A newer file shadowing the release's own. The module loader resolves a plain JavaScript file before its compiled form, and an update lays a new archive over the install directory without deleting anything. So a box could report the new version and run older code. All 18 route modules were affected. It is a supply-chain shape as much as an update bug — it is the cheapest way to swap out a piece of a protected panel on somebody's server. The first version of the sweep that was meant to fix it removed nothing on a box holding 62 shadowed files, because a substring match made every shadow look like a shipped file. The corrected sweep counts: 62 before, 62 removed, 0 after.

Integrity, and the panel's own exposure

Three read-only answers now exist to "is this still the software I installed": a build-time manifest for the panel's own files, a git-plus-stat baseline for the application (21,777 files under the dependency tree unchanged since the last deploy, bounded to 900 ms), and a write-time ledger over the 37 configuration files the panel writes — a hand edit to one was reported and the file restored.

The panel's own front door was closed in the same round, and the reason it needed closing is the reusable part. Direct access by IP and port:

The host header was the only thing the lockdown checked, and the caller writes that header. The peer decides now. Worse, the check that was supposed to close the door ran about 40 seconds after the DNS record was created, failed on a clean install where nothing was actually wrong, left raw access on — and nothing re-ran it. That box would have served its whole life with the panel reachable on its raw address.

The brute-force ban service earned its place quickly: one address banned within four hours, on a box that had existed for half a day and whose address had never been published. Two other ban rules were deliberately not shipped, because both see addresses arriving through a CDN, and a kernel-level ban there could blackhole an entire edge range and take the whole shop off the internet.

Below the web server, kernel network settings went from 1 to 13, the device backlog from 1,000 to 16,384, the half-open queue from 1,024 to 8,192, the panel's descriptor limit from 1,024 to 65,535, and its concurrent connection ceiling from none to 512. The accept queue was already at its 4,096 default and is now named and checked rather than left implicit.

Every one of those is a queue or a timer. Nothing there decides whether a packet is allowed, so nothing there can lock an owner out of their own server — filtering belongs at the provider's edge, where dropped packets never reach the machine at all.

A backup is only proved by a restore

Backups reported success while containing no database, which is why this is measured by reading the artifact rather than the exit code.

One full backup of a live box with the real dataset: 10,428 files and 903.0 MiB processed, 820.2 MiB added, 16 seconds wall clock, three database dumps with the largest 39.1 MB compressed. Read back from the repository, a snapshot contains the certificates, the rendered site definitions, the server's own settings, the panel's records, the shop's database dump, the shop's settings, its uploads, and the API signing keypair.

The restore was verified by row count, not by a status: 192 tables, 236 settings rows, 2 zones, 17,534 customers, 85 stores, 3,865 items, 66,701 orders.

Two defects came out of that check. One project was backing up its entire storage directory — logs, sessions, every compiled view — while another got uploads only. And both codebases keep the API signing keypair directly under storage rather than under the application's own subdirectory, so one project's backup never contained it and a restore would have signed out every phone.

Related

The aaPanel benchmark is where the isolation comparison sits beside the speed one · What we withdrew · How we measure · SixPanel vs SixPanel Docker vs aaPanel for the decision.