OpenLM.ai Leaderboard — with API pricing per million tokens (USD) — Last update: November 4, 2026
Data sources: openlm.ai/chatbot-arena (scores) & pricepertoken.com (prices), with OpenRouter as a fallback for prices and release dates, and Hugging Face for open-source status and Q4 size
⚠️ Not every LLM gets evaluated here — this table is OpenLM's top 100 by Elo, so a model simply isn't listed until OpenLM has scored it. Don't be surprised to see well-known names missing, e.g. Laguna S 2.1, Laguna XS 2.1, Qwen3.6, or Gemini 3.5 Flash Lite (see the Local Inference tab's "Laguna & Qwen3.6 remark" for more on that).
| # | Model ▾ | Elo ▾ | Coding ▾ | Vision ▾ | PP $ ▾ | TG $ ▾ | Score % ▾ | QP/PP ▾ | QP/TG ▾ | Release Date ▾ | Open-source ▾ | Q4 size ▾ |
|---|
🏠 Why local inference?
💵 Price: using many AI agents to automate many tasks can quickly become expensive in cloud AI tokens — running models locally removes the per-token cost, only hardware and electricity remain.
🔒 Sovereignty: sensitive data processed by a local AI never ends up in the logs of a US or Chinese cloud AI's servers.
📖 PP & TG — the two phases of inference
Every LLM response happens in two phases. PP (prompt processing, aka prefill, aka Input) is the input side: the model reads the whole prompt at once and pre-computes the KV cache for every token — highly parallel, fast work, measured in tokens/s (typically hundreds to thousands). TG (token generation, aka answer, aka OUTPUT) is the output side: the model produces the answer one token at a time — slower, because each token requires re-reading the whole model and its KV cache from memory, measured in tokens/s (typically tens). PP decides how long you wait for the first word on a long prompt; TG decides how fast the rest of the answer flows.
🖥️ Hardware recommendations to run a local AI inference
3 performance bottlenecks for local inference:
🚫 Why no table below 32 GB VRAM?
The reason is the intelligence of the LLM that can run locally. Below 32 GB you are stuck with tiny, not very smart models. Sure, a small LLM on an RTX 5060 (16 GB of GDDR7) would run extremely fast thanks to the memory bandwidth and the powerful GPU — but a stupid model that runs fast is still a stupid model. Either you have to compensate with a huge number of prompts to correct its errors, or the AI agent powered by this small LLM will be very fast at doing stupid things on whatever infrastructure, application or data it has access to.
⚖️ Unified memory vs gaming GPU
A machine with 64 GB or more of unified memory (Apple Mac mini, Mac Studio, MacBook, AMD Strix Halo, AMD Gorgon Halo) is often a better choice for local inference than a giant gaming rig with a beastly RTX 5090. Sure, the RTX 5090 has ~4× the memory bandwidth and a vastly faster GPU, so it would run an LLM much faster than the Mac — but only a smaller, less capable LLM. That is exactly the 1st bottleneck: VRAM size decides which model you can run at all; speed only decides how fast it runs. Local inference is a delicate balance between intelligence and speed. Running a dense model of 120B parameters on a 128GB VRAM computer will put your patience to the test. Running a small LLM of 12B parameters will be very speedy and not very smart, resulting in up to 10 times more prompts to achieve the same result as a smarter model.
📏 Context size
Context size: a large context is great — the LLM keeps track of its ideas and keeps working on its project without forgetting anything. But prompt processing (PP) slows down considerably as the context fills up; that is exactly where the KV cache matters, to increase the cache hit rate. A large context therefore demands a serious GPU/NPU (lots of TOPS) and extra memory bandwidth. Example: on an RTX 5090 you can run Gemma-4-26B-A4B-it at its full 256k context, but on a MacBook with 32 GB of unified memory it is better to cap the context at 64–128k.
🗄️ What is the KV cache?
For every token processed, each attention layer stores a Key and a Value vector so future tokens can attend back to it without recomputing it — that stored set is the KV cache ("KV" = Key/Value). It grows linearly with context length and is allocated up front for the full --ctx-size, not just what a prompt actually uses, so a model started with a 256k context reserves 256k-worth of KV cache immediately, even on the first, short request. Its size depends on the architecture, not just parameter count: layer count, number of KV heads and head dimension all matter, and techniques like GQA (fewer KV heads shared across query heads) or sliding-window/iSWA attention (only a handful of layers attend to the full context, the rest capped to a small local window) can shrink it by an order of magnitude or more versus a naive full-attention estimate. The "KV cache" column in the tables below is measured this way, per model, not calculated from a generic formula — see the footnotes under each table. One thing worth stressing: getting the KV cache 100% offloaded into VRAM, processed entirely by the GPU, matters just as much as doing that for the model weights themselves (see the "VRAM size" bottleneck above) — every generated token has to read the whole KV cache, so if any part of it spills into system RAM the GPU stalls waiting on much slower CPU memory, same as when weights spill. This isn't a minor slowdown either: if the model or the KV cache isn't 100% offloaded to VRAM, performance collapses dramatically for both PP (prompt processing) and TG (token generation) — not just one of them — because both need the full weights and the full KV cache on every pass, so a partial CPU fallback taxes every single step, not just the slow first one. The "Total VRAM" column below is model + KV cache added together for exactly this reason — that combined figure, not the model size alone, is what actually needs to fit on the GPU.
🧩 LLM architecture
MoE tip: for local inference on modest hardware, a Mixture-of-Experts (MoE) model is often preferable to a dense one — only a few experts are activated per token (e.g. 8B active of 118B total for Laguna S 2.1), so generation stays fast while quality rivals much larger dense models. Trade-off: the full weights still have to fit in VRAM.
🖼️ Multimodal
A multimodal LLM can process other modalities besides text — typically images, audio or video. The most useful in practice is vision: the model can read screenshots, diagrams, documents or UI captures, which is very handy for AI agents. Trade-off: multimodal models need a vision encoder (extra VRAM) and are slower on image input; text-only models are leaner and faster. The multimodal models in the tables can also be used in text-only mode.
🔢 Quantization
Quantization tip: BF16 is almost always overkill — Q8 (8-bit) is very close in precision, while the file size is halved and inference performance is nearly doubled. Q4 (4-bit) — the default used in the tables below — halves the size again and roughly doubles speed once more, with a small but real quality drop; it's usually the right call when VRAM is the binding constraint, while Q8 is worth the extra room when you have VRAM to spare.
🔀 Parallel slots
Multi-slot: with enough VRAM you can run several agent instances in parallel (llama-server --parallel N) instead of serial calls — e.g. ~3× Gemma-4-31B-it Q8 or ~4× Gemma-4-26B-A4B-it Q8 simultaneously on 128 GB (with reduced context). Parallelizing is very interesting: each agent keeps its own KV cache, so nothing has to be flushed when switching between agents — contexts persist, the effective cache grows, and prompt processing (PP) improves because the prompt is not re-processed from scratch on every switch. Most rows below show single-slot (Slots = 1) sizing as the baseline — a couple in the 32 GB table show a real 2-slot example instead — but the principle scales the same way: multiply the per-slot KV cache figure by the slot count and add it to the model size, and stay within your VRAM budget.
⚡ Speculative & predictive decoding (MTP, DFlash, DSpark)
Token generation (TG) can go well beyond the memory-bandwidth limit of bottleneck #2 — with zero quality loss — thanks to speculative decoding: a cheap drafter proposes several candidate tokens and the main model verifies them all in a single forward pass, keeping the accepted ones. Because every draft token is still checked by the main model, the output distribution is identical to normal generation — a pure speed trick. Two families exist: MTP (multi-token prediction), a small extra head trained with the model (Qwen3, DeepSeek-V3) that predicts the next N tokens; and separate draft models — DFlash (e.g. Poolside's Laguna drafts) and DSpark (DeepSeek's confidence-scheduled variant, whose V4-Flash drafter ships attached to the weights, no extra file needed). Measured on a Strix Halo machine: Qwen3.8-27B + MTP jumped from 27 to 39 tok/s (+45%, ~93% draft acceptance); Laguna S 2.1 + DFlash from 22 to 34 tok/s (+54%). Practical notes: the drafter costs a little extra VRAM; the gain depends on the acceptance rate, so tuning --spec-draft-n-max matters (4 is often the sweet spot, not 16); some models embed the drafter, others need a small separate GGUF file.
⭐ August's LLM stars
Three open-weight models made August remarkable — each one strikingly intelligent for its weight, and all three come with a speculative/predictive mode: they run comfortably on 128 GB with multiple parallel slots (on 64 GB with fewer slots and/or less context — Laguna needs a lighter quant like IQ4_XS there).
⚠️ Recommendations valid at a point in time
The recommendations in the 32 / 64 / 128 GB tables below are valid at a point in time — new, interesting models are released fast (Qwen3.8 is a recent example), so these tables require frequent updates.
Hardware recommendations to run a local AI inference: any GPU with 32 GiB of VRAM like an RTX 5090, or a computer with 32 GB of unified memory (Apple Mac mini, Mac Studio, MacBook or a Strix Halo AMD platform) — all three models below fit at their full listed 256k context in Q4, live-measured with room to spare; Q8 also fits for Gemma-4-26B-A4B-it with a short context.
| # | Models sorted by Intelligence on performance ratio | Architecture | Multimodal | Max Context | Size | KV cache size | Slots | Total VRAM | Arena Elo | Coding |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Qwen3.6-35B-A3B | MoE | Multimodal | 256k | 20.6 GiB (Q4) | 4.2 GiB | 2 | 29.0 GiB | — | — |
| 2 | Gemma-4-26B-A4B-it | MoE | Multimodal | 256k | 15.8 GiB (Q4) | 5.3 GiB | 2 | 26.4 GiB | 1,443 | 1,460 |
| 3 | Laguna XS 2.1 (APEX I-Compact) | MoE | Text only | 256k | 14.7 GiB (Q4) | 10.3 GiB | 1 | 25.0 GiB | — | — |
Hardware recommendations to run a local AI inference: 64 GB class (e.g. RTX 6000 Ada, dual 32 GB GPUs, M-series Ultra, or 96 GiB unified memory like the Strix Halo) — Q8 becomes the sweet spot for ~30B models, Q4 for larger MoEs.
* Gemma-4-31B-it's, Gemma-4-26B-A4B-it's and Qwen3.6-35B-A3B's KV cache figures are live-measured (each loaded standalone at 262144 ctx, 1 slot, GTT delta via amd-smi minus known weight file size) rather than calculated. Gemma uses iSWA attention (5 global / 25 sliding-window layers); Qwen3.6's real footprint also came in far below a naive full-attention estimate. All three figures fold in llama.cpp's compute buffers alongside KV cache, since per-component logs weren't captured.
| # | Models sorted by Intelligence on performance ratio | Architecture | Multimodal | Max Context | Size | KV cache size | Slots | Total VRAM | Arena Elo | Coding |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Qwen3.6-35B-A3B | MoE | Multimodal | 256k | 34.4 GiB (Q8) | 4.2 GiB* | 4 | 51.2 GiB | — | — |
| 2 | Gemma-4-26B-A4B-it | MoE | Multimodal | 256k | 25.5 GiB (Q8) | 5.3 GiB* | 4 | 46.7 GiB | 1,443 | 1,460 |
| 3 | Laguna XS 2.1 (APEX I-Balanced) | MoE | Text only | 256k | 22.6 GiB (Q5) | 10.3 GiB | 3 | 53.6 GiB | — | — |
| 4 | Gemma-4-31B-it | Dense | Multimodal | 256k | 30.9 GiB (Q8) | 21.7 GiB* | 1 | 52.6 GiB | 1,449 | 1,462 |
Hardware recommendations to run a local AI inference: 128 GB unified memory or dual 64 GB GPUs — plenty of room for Q8 multi-slot agents (llama-server --parallel), or Q4 models up to ~110 GB.
* Gemma-4-31B-it's and Laguna S 2.1's KV cache figures are live-measured (each loaded standalone at 262144 ctx, 1 slot, GTT delta via amd-smi minus known weight file size) rather than calculated. Gemma uses iSWA attention (5 global / 25 sliding-window layers), which a naive full-attention formula wildly overestimates. Both figures fold in llama.cpp's compute buffers alongside KV cache, since per-component logs weren't captured.
‡ Laguna S 2.1's real max context is 1M, confirmed via its own GGUF metadata (the "256k" shown is the practical multi-slot operating point — 68.1 GiB weights (Q4_K_M, unsloth) + 12.2 GiB KV per slot → 4 slots = 116.9 GiB total, comfortably fit 128 GB). At Q4 the full 1M context also fits single-slot (extrapolated from real 256k/512k measurements, ~116.5 GiB total — at Q5_K_XL it was ~130.4 GiB, over budget); 256k is shown because it allows 4 parallel slots for the same VRAM.
| # | Models sorted by Intelligence on performance ratio | Architecture | Multimodal | Max Context | Size | KV cache size | Slots | Total VRAM | Arena Elo | Coding |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Laguna S 2.1 | MoE | Text only | 256k (up to 1M)‡ | 68.1 GiB (Q4) | 12.2 GiB* | 4 | 116.9 GiB | — | — |
| 2 | Gemma-4-31B-it | Dense | Multimodal | 256k | 30.9 GiB (Q8) | 21.7 GiB* | 4 | 117.7 GiB | 1,449 | 1,462 |
A high-level, step-by-step tutorial — no prior infra experience required
🎯 What makes an assistant "personal"?
Not just a chatbot tab you open when you remember to. A personal AI assistant is always reachable (on a channel you already check, like a messaging app), has memory that survives across conversations (it doesn't forget who you are every time), can act on your behalf (run tasks, check things, use tools — not just talk), and can be triggered on a schedule, not only when you type to it. Everything below is one piece of that picture; you can build it incrementally.
💸 You don't need expensive hardware to start
This is the most important thing to get right before spending a cent: local inference (the Local Inference tab) is an optional upgrade, not a prerequisite. The cheapest, fastest way to get a working personal assistant running today is a cloud API model — pay per token, zero hardware, working in an afternoon. Treat this as a deliberate evaluation phase, not a compromise: there's a wide field of cheap, genuinely capable cloud models built exactly for this kind of everyday assistant workload (chat, quick lookups, light agentic tool use), and running a few of them for a couple of weeks is the fastest way to find out whether a personal AI assistant actually earns its keep in your day-to-day life at all — before you decide anything about hardware. Shop for that trial on the Chatbot Arena tab: sort by "QP/TG" or "QP/PP" (quality per dollar) rather than raw Elo, and several strong options come out at fractions of a cent per request. Only once you're using it daily and can actually see the token cost adding up — or you specifically want data sovereignty (see the Local Inference tab's "Why local inference?" section) — does buying VRAM start to pay for itself. Building on cloud first also de-risks everything downstream: you validate the agent framework, the channel, the memory design and the automation before ever touching a hardware decision.
It's not a free lunch, though — it's a trade, not a shortcut: going cloud swaps the Local Inference tab's 3 hardware bottlenecks (VRAM size, memory bandwidth, GPU/NPU power) for 3 different things to watch instead:
🛡️ Confidentiality & privacy: think before you paste
Before wiring a cloud model into anything real, be clear about what it is: a third-party service run by a foreign company, and you have no visibility into what it actually logs, retains, or reuses. Never process genuinely sensitive data through it, and never hand it live credentials — SSH private keys, API keys, database passwords, cloud provider secrets, or anything else that grants real access to IT resources that matter. If your assistant needs that kind of access to do its job, that's precisely the workload that belongs behind local inference instead (see the Local Inference tab), where nothing leaves your machine. This matters even more when the model is free rather than pay-per-use: a paid provider's incentive is to keep your trust so you keep paying, but a free tier's business model is very often built on your traffic itself — logs, prompts, conversations — being the actual product. Free is rarely free; the data you send may be the payment. (This is separate from Step 7 below, which covers securing the assistant you build — this is about what you choose to send to someone else's model in the first place.)
🤝 Step 1 — Choose an agent framework
A raw chat model can only talk. An agent framework is the loop around it that lets it act: it defines tools (read a file, run a command, call an API, search the web), feeds their results back to the model, and repeats until the task is done. This is what turns "answers questions" into "does things." Pick one that supports your model's tool-calling format and can run as a persistent background service rather than a one-shot script — you'll want it always running, not just open in a terminal.
Two names come up constantly: OpenClaw and Hermes. OpenClaw is the more established, historically first-mover framework, and its real strength is ecosystem size — a huge community-built library of skills, MCP servers and tool integrations, so there's a good chance someone has already built what you need. That size cuts both ways: a community-sourced skill/MCP library also includes a nonzero amount of outright malware, and vetting anything before installing it is essential — a topic that deserves its own dedicated write-up, not a footnote here. Hermes is this page's own pick, for two concrete reasons: it's Python end-to-end, no Node.js/npm toolchain to babysit alongside it, and its standout feature is self-improvement — it automatically keeps its own notes on what it did and why, and turns recurring work into reusable skills on its own, so it doesn't reinvent the same solution from scratch the next time a similar task comes up. Neither is objectively "correct": OpenClaw if you want the biggest available tool ecosystem and are prepared to vet community code yourself, Hermes if you want a leaner, self-documenting, Python-native footprint.
Recommendation: Hermes, for its automatic self-improvement feature — it keeps its own notes and turns what it did into a reusable skill without being asked. OpenClaw has no equivalent built in: left to itself, it's functionally amnesiac, and without a reminder every 45 minutes or so to write down what it just did and turn it into a skill, it just won't remember how it solved the same problem next time.
🖥️ Step 2 — Where to install the agent
Don't confuse this with the model's hardware needs (Local Inference tab) — the agent framework itself is not what needs the beefy hardware, even if you go local. Its own job is orchestration: run the conversation loop, call whatever model you configured (cloud or local), run tools, talk to your channel. On its own, that footprint is ridiculously modest — a headless Linux server with a dual-core CPU and 2 GB of RAM is good enough to run it comfortably (a rule-of-thumb figure, not a benchmarked one — worth confirming against your specific framework's real footprint once it's installed, but that's the right ballpark to plan around). That means almost anything you already have lying around will do: a small mini-PC, a Raspberry Pi, an LXC container, or a small VM all fit perfectly — no need to dedicate your main workstation or buy anything new just to host the agent process itself.
Recommendation: an LXC on a Proxmox hypervisor, running Ubuntu Server 26.04 LTS, fully updated — lightweight, easy to snapshot/back up before changes, and easy to resize later if the modest footprint above turns out to be wrong for your setup.
⚙️ Step 3 — Choose an inference layer
This is the cloud-vs-local decision, and it doesn't need to be permanent — see "You don't need expensive hardware to start" above. Cloud: point your agent framework (Step 1) at an API key and an HTTP endpoint — most frameworks speak the OpenAI-compatible chat completions format natively, so there's nothing to host yourself. Local: you need a server that loads a GGUF file and exposes that same OpenAI-compatible API — llama.cpp's own llama-server is the reference implementation. If you expect to run more than one local model (e.g. a fast small one and a smarter big one), a swapping proxy in front of it (like llama-swap) will load whichever model a request asks for on demand and unload idle ones after a timeout — much better than running every model resident 24/7.
Recommendation: OpenRouter for the cloud path — it's prepaid credit, not a monthly subscription, so there's no recurring commitment while you're still in the evaluation phase, and one API key/endpoint gives you access to most of the models on the Chatbot Arena tab.
🧠 Step 4 — Choose a model
Now that the framework and the hosting decision are made: on OpenRouter (or any other cloud endpoint), pick anything from the Chatbot Arena tab with good "Score %" and a price you're comfortable with — you can switch models later with a one-line config change, so don't overthink this first pick. Going local instead: pick a model that fits your VRAM tier on the Local Inference tab's tables — that's what those tables, and the KV cache explainer above them, are for.
Recommendation: start with OpenRouter's Free Models Router (openrouter/free) — a virtual model that automatically routes each request to a free model capable of handling it. The reason this matters in practice, not just in theory: free models carry aggressive rate limiting, and after a handful of queries a given one will simply stop responding and tell you to wait — the router's whole job is to fail over to a different free model when that happens, so your assistant keeps working instead of stalling. Zero cost while you're still validating the framework, channel and memory pieces below. Two real trade-offs to know about, though: you don't actually know which underlying model answered any given request, which matters if you care about data sovereignty (see the Local Inference tab's "Why local inference?" section), and free models are not equally smart or efficient — response quality can swing noticeably from one query to the next depending on which one you were routed to. Graduate to a specific paid model, or a specific local one, once you know the assistant is worth keeping and consistency starts to matter.
Secondary recommendation: if the free models' rate limiting gets hit quickly for your usage, move to Gemini 3.5 Flash Lite — cheap enough that daily assistant use stays close to pocket change, capable enough for everyday chat and light agentic tool use, and without the free tier's throttling or model-uncertainty trade-offs. Check its live price and score on the Chatbot Arena tab.
💬 Step 5 — Give it a channel
A messaging app you already have open beats a dedicated web UI you have to remember to visit — most agent frameworks support Telegram, Discord, Slack, or plain SMS bridges out of the box. Whichever you pick, restrict who can message it (an allowlist of your own account ID) before it goes live; an open bot is an open door.
Recommendation: start with Telegram — the simplest bot setup of the bunch, good enough to get talking to your assistant in minutes. Discord is worth adding later, once the project grows: channels/threads for organizing different areas of the assistant's work, built-in screen sharing, and built-in voice make it a better fit once you're past the initial "does this even work" phase.
🗂️ Step 6 — Give it memory
This is about long-term memory — facts, preferences and past decisions that need to survive after a conversation ends and its context window is gone (the conversation's own short-term context is handled automatically by the model itself, nothing to build there). It doesn't need to be fancy at first: plain text or markdown notes the assistant reads before responding and writes to when it learns something durable work fine, and are easy to inspect and correct by hand. Reach for a vector database only once plain files stop scaling.
Storage sizing: 512 MB is plenty to start — notes and a self-improving framework's skill files are tiny on their own. But don't be surprised when that stops being enough: an actively-used assistant piles up scripts, generated documents, downloaded reference material and its own working files fast, and 1 TB becomes a realistic real-world figure sooner than you'd expect. Storage is cheap; plan for growth rather than sizing tightly up front.
⏰ Step 7 — Automate it
A real assistant does things without being asked: a scheduled (cron-style) trigger that runs a prompt on a timer turns your assistant from purely reactive into proactive — a daily summary, a periodic check, a reminder that fires itself. Most agent frameworks have a scheduling feature for exactly this; if yours doesn't, a plain OS-level cron job calling the same API your channel uses works just as well.
Observation, not a recommendation: this is not theoretical — the Chatbot Arena tab on this very site is refreshed daily by a personal AI assistant's own scheduled job, unattended.
🔒 Step 8 — Lock it down
An assistant that can run commands or read files is a real access point, not a toy — treat it accordingly. Never put API keys or passwords directly in the prompts, memory files or notes it reads and writes; keep secrets in environment variables or a proper secrets store instead. Restrict who can talk to it (see Step 5). If it exposes any local network port (its inference API, a dashboard, anything), put it behind authentication before exposing it beyond your own machine — never assume a private IP or an obscure port is enough on its own.
🧭 Common pitfalls
Sizing local hardware from a spec sheet instead of a real measurement — KV cache in particular is very architecture-dependent and generic formulas are frequently off by 5-10× (see the KV cache section on the Local Inference tab); always verify with a real load once it matters. Running several local models resident at once "just in case" instead of swapping on demand — wasted VRAM and wasted idle power. Skipping the memory layer at first and being surprised the assistant "forgets everything" — that's expected until Step 6 is built. And treating the first model/framework/channel choice as permanent — every piece here is swappable independently; start simple, cloud-first, and upgrade the pieces that actually turn out to matter for how you use it.