I’ve been running local AI models for almost a year. Through the course of learning, deploying and tweaking my own robust agentic system, while staying abreast of ever-accelerating developments in this field, I’ve developed a singular allergy to benchmarks.
Benchmarks are useless. There, I said it.
When I question real users concerning model selection, they usually sling a generated graph and a link to a major benchmarking website: MMLU, GPQA, Arena ratings. The numbers look impressive, but none of them answer the question I actually asked.
My question is always about real-world utility: will this model do the work I put in front of it, at a speed I can live with? Don’t give me sterile, universal benchmarks, I want real user-accepted results.
Currently my daily assistant runs on a local model. Most times results are on-target, meet my expectations, but occasionally it sets off on a cascade of bad decisions and things just fail silently. A model that tops a leaderboard is worthless the moment it can’t hold up under the tasks I actually run and need, which is the only point of SI for me.
Benchmarks test models on suites of universalized problems meant to challenge an LLM enough to result in failure a percentage of runs. You need work; the benchmarks need failures.
The Tradeoffs
As the simplest polygon, the triangle is the most basic version of a radar (spider) chart used to represent allocation. In determining the most “subjectively resourceful” model, we can plot our three values from the center across equal radii. A tradeoff radar chart acknowledges that you have a finite or fixed resource, and plotting one longer radius results in a reduction across one or more other radii.
You may be familiar with a few tradeoffs:
- In modern photography: user-acceptable images have an adequate combination of Aperture, Shutter Speed, and ISO as levers to manipulate light reaching the sensor. Tweaking radii results in more or less depth of field, blur, and/or noise to fit user taste.
- In employment and professional services: the acceptability of work will almost always be a tradeoff between cost, time and quality.
The comparison that matters for us here also has three spokes which pull against each other:
- Speed: token prefill & generation, measured separately, because a model can eat up context quickly and write the answer glacially slow.
- Capability: most comparisons I’ve seen skip capability discussion in favor of the vanilla, failure-driven benchmarks. User-accepted capability means output you actually keep, and needs to drive the conversation.
- Size: parameter count and quantization. These together decide what your hardware can even load.
There’s a reading-speed argument that gets raised about generation speed. A human can read on average about 238 words per minute, which converts to about 5 tokens per second. My current 44 tok/s should be at about 8 times human read speed in basic chat, but this all goes out the window with Agentic, work-facing scenarios. When the tool calls and executions line up, the LLM is processing walls of code, but as the user, it’s like watching paint dry. So for our purposes, speed is “how long until the whole task finishes.”
The Triangle
Here’s the shape of the problem: you have a limited resource and can optimize any of the axes, always at the cost of one or more of the others. Hence the triangle, and the constraints every local LLM user faces.
The limited resource: your hardware. RAM size and processing speed are hard limits; they give you what they can. The model you end up running is always a compromise between how many parameters it carries, how precisely those parameters are stored, and how fast it moves.
My own setup is a useful contrast to most of the advice out there. I run dual 32 GB RX 9070s on a Threadripper TRX-40 board with 64 GB of system RAM, and room to grow. So while most people have to compromise on model size to fit whatever memory they have, new developments in training, models and harnesses allow me to dabble with the larger models at higher precision and see what the compromise actually costs. I should be happy with what I have, but the more time I spend with LLM work, the more I demand higher-quality, faster results, and the only way to reduce tradeoffs is to increase the fixed resource.
Capability In Parameters. Accuracy In Quantization.
It’s a common mistake in local model shopping to treat size as one number: what can fully fit on my GPU RAM. While it’s hard to conceptualize, parameter count does symbolize capability or what the model can do, and those numbers themselves, 9b, 27b, 125b, really do mean how happy you will be with the results. Quantization on the other hand, is a compromise on accuracy, how faithfully the model reaches that result.
A parameter count tells you what a model can do. A quantization tells you how accurately it does it.
Allegorically, if we were to say 27b model can play soccer, a 125b model might be able to play multiple sports, and a full, un-quantized version scores goals 100% of the time. There’s a reason why users jokingly refer to low-quant models “lobotomized.”
Regarding quantization, I’ve found this first-hand: in running Qwen 3.6 and 3.8 at 27B for my agentic work, at anything below Q6 the model still gives you a plausible-looking answer, but struggles to make good tool call decisions. A lower quant delivers poor results in exactly the spot that matters to me, which of course did not match at all what the model leaderboards told me.
My standard, then, is to fit on my hardware the highest parameter count with the most reliable Unsplash quant which produces acceptable results. I accept the consequences: a smaller context window and slower answers… for now. A stellar final result is always a better trade.
“Q4” Is Not the Same on Two Models
As I’ve grown my system over the past months, I have been delighted by the accelerating cadence of model, harness and software drops. I went through my Qwen 3.6 phase and am currently all-in on Qwen3.8-27B. I tried flavors of “Q4” ggml-org, Unsloth quants, and landed on Q6 due to the accuracy, or certainty, of the results.
You can imagine that when I see people running Q2 quants “at high speeds” with the model I’m actually investigating, Qwen3.8-Flash-Next, I am a bit skeptical. This is a different architecture: 125B total parameters with 6B active, plus a 51B n-gram embedding layer. My gut says the quantization will affect the user-acceptable results far less than a 27b model, so the open question I’m sitting on right now is what the quality floor looks like for Flash-Next. Is Q3 enough for an MoE with that architecture, or do I push for Q4 or even Q6 and take the context window hit again? I don’t know yet. I’ll find out by running it.
The Bigger Picture
Local LLMs keep moving. With Strata deployment strategy gaining adoption and Radiance server tuned for AMD GPUs, as well as dense 27B models being superseded by larger MoE hybrids, the hardware equation seems to change almost daily. The tradeoff triangle remains: whatever the next model is, you will still be trading speed against size against the accuracy of the output, and the method that works is the one I described: test what fits on your hardware, set the speed floor you can live with, then take the highest-fidelity version that still meets both and delivers an acceptable result consistently. That method outlasts any single model.
Related Reading
- You’re Going to Need to De-Google. Let Me Tell You Why.
- Shodan: The Search Engine for Exposed Devices
Brendon Brown: I am a fractional CTO and digital strategist working with private brands, religious institutions, and mid-market businesses that refuse to settle for mediocre technology and user experiences. I learn new software quickly, push it to its limit, and build bespoke pieces to fill clear gaps. Fourteen years in digital marketing, IT infrastructure, and eCommerce migrations have taught me that most companies are running on systems that actively work against them: bloated, expensive, badly integrated, and genuinely ugly. Let me fix that.
Running local models and staring at the same triangle? The hardest part isn’t picking a model; rather, it’s knowing which of the three you can afford to give up. Let’s just talk for thirty minutes.