Local LLMs are becoming a workflow choice, not a hobby project

Jamesob’s guide to running SOTA LLMs locally hits a nerve because the local model conversation has changed.

A year or two ago, local inference was mostly a tinkerer’s victory lap. Can I get this thing to load? Can I make tokens stream on a consumer GPU? Can I avoid melting my laptop?

Now the better question is: what work should not leave my machine in the first place?

That shift matters. Local LLMs are not replacing the best hosted frontier systems across the board. They do not need to. The useful frame is narrower and more operational: local models are becoming good enough for a growing set of private, repetitive, latency-sensitive, and cost-sensitive tasks.

“SOTA locally” is a relative claim

SOTA is doing a lot of work here.

When a hosted lab says frontier, it usually means the best general model they can serve at scale, with huge training runs, productized tool use, safety layers, and managed infrastructure behind it. When builders say SOTA locally, they usually mean the best open or open-weight model that can run acceptably on their hardware after quantization, memory tradeoffs, and context compromises.

Those are different contests.

That does not make the local path fake. It just makes it specific. A quantized model on a workstation can be excellent for codebase Q&A, document cleanup, meeting note restructuring, local search, test generation, classification, and first-pass drafting. It may be weak on long-horizon reasoning, messy agent loops, brittle tool calls, or tasks where the hosted model’s hidden scaffolding is doing more than the raw model.

This is the part that gets lost in hype cycles. Local is not a badge of purity. It is an architecture choice.

The bottleneck moved from install to judgment

Guides like Jamesob’s are useful because installation is no longer the whole game. The stack has matured enough that many builders can get something running with tools such as llama.cpp, Ollama, LM Studio, MLX, or vLLM, depending on hardware and taste.

The harder work starts after the first successful prompt.

Which model is good enough for this workflow? Which quantization keeps the behavior you need? How much context do you actually need, not just want? Does the local model follow your output schema every time? Does it cite the right chunks from a local retrieval system, or does it confidently blend two files into one imaginary fact?

Local models make these questions more visible because there is less product padding. You feel the tradeoffs directly: VRAM, RAM, disk, heat, latency, context length, batching, and failure modes.

That is healthy. Hosted APIs hide a lot of complexity, which is one reason they are useful. Local systems expose complexity, which is one reason they teach you what your application actually depends on.

The winning setup is probably hybrid

I do not think the interesting future is “all local” or “all cloud.” It is routing.

Run sensitive files locally. Run cheap repetitive transforms locally. Run embeddings or reranking locally when privacy matters. Send the hard cases, high-stakes reasoning, broad web questions, and gnarly multimodal work to a hosted model when the quality gap justifies it.

That design also protects you from model churn. If your app has a thin model adapter, basic evals, and clear task boundaries, you can swap a local model in for one class of jobs and keep hosted inference for another. If your app is just one giant prompt glued to one provider, you have less room to move.

For a builder, the useful experiment is simple: pick one workflow you run often, ideally one with private or repetitive input, and make a local version of it. Measure boring things: time to first useful answer, correction rate, schema failures, cost avoided, and whether you trust the output enough to use it without babysitting. The catch most readers miss is that local LLMs do not remove operations work. They move it closer to you. That is either the point, or the reason to keep paying an API.