# Local AI: open models on this server > Source: https://www.allsweb.com/sixpanel/docs/local-ai-models > Markdown for agents: https://www.allsweb.com/sixpanel/docs/local-ai-models.md > Publisher: AllsWeb (www.allsweb.com) Part of: SixPanel documentation **What this page is for:** run an open AI model — Qwen, Gemma, Mistral, DeepSeek or GLM — on this server itself, and let Tia and AI monitoring use it. Nothing you ask it leaves the server, and there is no provider to pay. **The rule that does not move:** your shops and websites come first. Local AI only gets the memory and the CPU they do not need, it answers one question at a time, and it is the first thing the server stops if memory ever runs out — never a shop. **You need** - An **amd64 or arm64** server with **at least two CPU cores** (one is always left to your websites) on Ubuntu 26.04, Ubuntu 24.04 or Debian 13. - Enough **free memory** after everything else — the page works it out for you (below). As a rough guide: a small model for tests needs about 1 GB to spare; a model that can answer Tia needs about 1.5–2.5 GB at the context she needs; the larger ones 4–25 GB. An 8 GB server with one shop can give a model a little over 1 GB. - About **1.5 GB of disk** for a moment while Ollama is installed, with 2 GB more left free beside it (about 80 MB is kept), plus each model's own size. - The owner's login to install, download, change or remove anything. A temporary login sees the page and changes nothing. - **SixPanel Pro** to ask a model anything — the test question, Tia and monitoring — like every AI request in the panel. Installing and downloading are free. --- ## Where it is **Software → Local AI.** On **AI → Providers**, under the list, **Local models** says whether this server can run one — and if it cannot, how much memory it would need — with the way to it. ## What this server can give a model The top of the page says it in one line — on a 16 GB server with one shop and 450 MB of data in its database: *This server can give a model 10.4 GB of its 15.6 GB of memory, and 7 of its 8 CPU cores.* **How this is worked out** opens the sum: | | | |---|---| | **Memory of this server** | what the server has | | **The system and SixPanel** | kept for the operating system and the panel | | **The database** | what MariaDB uses now, and what your databases' data can still fill of its cache | | **Redis** | what every Redis on the server uses now — the shops' and the websites' own | | **PHP, at full load** | every PHP worker your shops may start (only with a shop) | | **The shops' storefronts and queues** | each 6amMart shop's storefront and its queue worker (only with a shop) | | **The websites, as measured** | what your websites use right now | | **PostgreSQL, the email server, Varnish, Memcached** | what each uses now, when it is installed | | **Docker and its containers** | what Docker and everything it runs use now — the Docker apps, and MongoDB and SQL Server, which are its containers | | **Kept free for a busy moment** | a margin — at least 256 MB | | **Left for a model** | what is left | | **Free right now** | what the server has free at this moment | What a service **uses** is what the server counts, not the most it may grow to: MariaDB's cache, for one, is sized at almost half the server's memory, and only your data fills it — so what is kept for it is what it uses and what your data could still fill. Each figure is the memory the service holds, without the file cache the server can drop at any moment (that is in *free right now* already), and each is counted once. Where the server cannot measure one, its most is counted instead. The PHP workers of your shops are kept at full load: a sale fills them in seconds. A model is given the **smaller** of *left for a model* and *free right now*. That is also its **memory ceiling**: Local AI cannot use more, and if it tries, it is the model that stops — never a shop. A second shop, more websites or a busier hour each make the number smaller; the page asks again every time it is opened, and the running Local AI is given the new ceiling right before its model is next asked (by Tia, monitoring or a test), without a restart. Its CPU is **all cores but one**, at the lowest priority the server has: when a shop or a website needs the CPU, it gets it first. A server that is resized to **one core** runs no model at all: Local AI is stopped when the panel next starts, and says why — its one core is your websites'. ## Install **Install Local AI** (on the page, or on the Software tab) does this, as a job you can watch in the drawer: 1. Checks that this server can run a model at all — and stops, saying why, if it cannot. That includes **another Ollama already answering on 127.0.0.1:11434** (Ollama's own `ollama.service` uses the same address, without any of these limits): Local AI never takes it over. Stop it first (`systemctl disable --now ollama`), then install. 2. Downloads **Ollama** from its own GitHub release (about 1.4 GB). 3. Checks the download against the **checksum this SixPanel release carries**. A file that is not exactly that build is refused and nothing is installed. 4. Unpacks only its **CPU part** (about 60–80 MB) — the GPU libraries are left out. 5. Gives it its own user, `sixpanel-ollama`, and its own folder, `/var/lib/sixpanel/ollama`. 6. Starts it as `sixpanel-ollama.service` on **127.0.0.1:11434** — this server only — with a rule that lets **only the panel** connect to it. Your websites cannot reach it. 7. Removes the download. No model is downloaded yet: you choose one next. When a SixPanel update brings a newer Ollama, the page says so with **Update to …**. The models and the settings stay. ## The models Every model on the list is shown with what it needs **on this server**, by family — **Qwen** (Alibaba), **GLM** (Z.ai), **DeepSeek**, **Gemma** (Google), **Mistral** and **Kimi** (Moonshot AI) — and by what it is good at. **Show** picks one category: | Category | What it means | |---|---| | **Fast** | small models that answer quickly on a CPU | | **Reasoning** | models that think a problem through | | **Coding** | models trained for code | | **Complex problem-solving** | the larger models, for harder questions | | **Chat** | general conversation | Each row says its download size and the **memory it needs** at your context (see Settings). The badges say a little more: **Tia** — it calls tools, so it can answer Tia; **Thinks** — it can reason before it answers; **Images** — it can read pictures. - **A model that fits** has **Download**. - **A model that does not fit** has no Download. It says what it needs instead: *Needs about 4 GB of memory; this server can give 3.4 GB. A server with about 24 GB of RAM runs it.* (A larger server gives most of its extra memory to the database first, so the size named is one that really runs it.) In each family the models that fit come first, and the larger ones are folded under them. - **Kimi, GLM-5, GLM-4.6 and DeepSeek-V3.2** are listed with the memory they would need (hundreds of gigabytes) and are **never offered**: Ollama runs them only on its own servers, as cloud models, and no server panel box holds them. The list ships with each SixPanel release and says the day it was read. Only models on the list can be downloaded — a typed name could fill the disk with a download nobody measured. ### Download **Download** asks first, then a short job checks that the model fits and asks Ollama for it — and ends. The download itself goes on **outside the job queue**, so a deploy, a rollback, a backup or a restore you start meanwhile runs at once instead of waiting behind gigabytes: the row shows a bar while it downloads, and the page says how it ended (a download that stopped says why and, unless you stopped Local AI yourself, you are told as for a failed job). **Stop download** beside the bar stops it; what came so far stays with Ollama, so **Download** later goes on from there. One model downloads at a time. A restart of the panel stops a download: press **Download** again. A download is refused at once — before any job — if the model does not fit, or if it would leave the disk without room for your shops (at least 2 GB, or 5 % of the disk, always stays free). ### Downloaded models The **Downloaded models** card lists them: size, whether it can answer Tia, and whether it is **in memory** now. - A model loads into memory when it is asked something and is **unloaded** by itself after a few idle minutes (Settings). **Unload now** frees the memory at once. - **Test** asks one question and shows the answer and how fast it came: *Loaded in 2.1 s · read 14 tokens at 120/s · answered 38 tokens at 9.5/s.* The first question after a pause includes loading the model. - **Remove** deletes the model's download. - **Use for Tia** makes Tia answer with that model (below). ## Use it for Tia Once Local AI is installed, **This server** appears on **AI → Providers** as a provider of its own, and it answers as soon as a model that calls tools (the **Tia** badge) is downloaded: - It answers **Tia and monitoring** — nothing else. It is never the default, and AI Coding and Create with AI never use it: they send far more than a small model on a CPU can read. - **Use for Tia** (here) or **AI → Settings → Use a different provider for Tia** chooses it. When no other provider can answer, Tia uses it by herself. - Tia needs a **context of 16,384 tokens** — her instructions and tools are about 11,000 tokens before your first word. **Monitoring** needs **32,768**. Until the context is large enough, This server is greyed out for them, with the reason. - Her answers come from this server's processor: slower than a provider's, and the first answer after a pause waits while the model loads. A model that can think is asked not to, so it answers sooner. - Its row on AI → Providers has **Check** and **Manage** — no Make default, no Edit, no Remove: Local AI itself is removed here. ## Settings | Setting | What it does | |---|---| | **Context** | how much a model reads at once: 4,096, 8,192, 16,384 or 32,768 tokens. Tia needs 16,384 and monitoring 32,768; more takes more memory | | **Unload after (minutes)** | a model nobody asks is taken out of memory after this long (1–60) | | **Memory saver** | how a model keeps what it has read: **Balanced** (recommended) halves that memory; **Off** uses the most; **Most** the least | | **CPU cores** | **Automatic** is all cores but one; you can give it fewer, never all | **Save** restarts Local AI with the new settings. A fresh install starts on the largest context at which a model that can answer Tia fits this server. ## Start, stop, remove **Stop** stops Local AI and keeps it stopped, also after the server restarts — it uses no memory at all then. **Start** and **Restart** write its service again from this server's numbers first. **Log** shows its newest lines. **Remove Local AI** removes Ollama, its service and its rule. The downloaded models stay, so installing again finds them — unless you tick **Also delete the downloaded models** and type `localai` to confirm. ## How it stays out of your shops' way - **One question at a time, one model in memory**, and a short queue: a crowd of questions is turned away, not piled up in memory. - **A memory ceiling** set from what this server can spare, with no swap — and it is the first program the server stops if memory runs out. - **All CPU cores but one**, at the lowest CPU priority. - **Its disk use held to 80 MB/s read and 40 MB/s written** on the disk its models are on — a ceiling the kernel keeps on every disk. It also asks for the lowest disk priority, but most cloud disks (NVMe, or a scheduler of `none` or `mq-deadline`) do not honour priorities, so the ceiling is the part that holds: on a disk faster than that, the rest is your shops'. A 4 GB model read from the disk therefore loads in about a minute. - **Unloaded when idle**: no memory used between conversations. - **This server only**: it listens on 127.0.0.1, and a rule lets only the panel connect — to its own port and to every port its model runners open while a model is loaded. It never uses Ollama's cloud models. - **Nothing runs until you install it**, and nothing is checked on a timer. **Server → Health** has a **Local AI** row: it fails if Local AI ever listens beyond this server (its model runners included), if its service lost these limits — read from what the server runs it with, so a drop-in added with `systemctl edit` cannot lift one unseen — if it runs without its rule, if another program answers on its address, or if it stopped by itself. ## If something goes wrong - **The page says this server cannot run it.** The sentence says why and how much memory it would need. A larger server, fewer shops or a smaller context changes the answer. - **A model is not offered.** It needs more memory than this server can give at your context. A smaller context, or the **Most** memory saver, may make it fit. - **Tia answers slowly.** Choose a smaller model that can answer Tia, and keep the context at 16,384. The first answer after a pause includes loading the model. - **This server is greyed out for Tia.** Download a model with the **Tia** badge, and set the context to at least 16,384. - **Local AI is not answering.** Press **Restart**. **Log** shows the newest lines of what it said while starting or loading a model. - **Ask Tia** "Is Local AI running, and which model fits this server?" — she reads the same page. ## The software card on this page The **AI** page shows the **Local AI** card once it is installed. It is the same card as on **Software**: its state, its version, a waiting update and, for the owner, Restart (asked first, and refused while another job runs). **On Software** opens its card there, with who else uses it. See [Software, search and the system monitor](https://www.allsweb.com/sixpanel/docs/software-search-monitor).