A private LLM is a large language model that runs on hardware you control: a server in your office, a box in your rack, or a dedicated machine in a private cloud. Nothing you send it crosses the public internet, no third party processes your prompts, and no usage meter runs.

Somewhere in most AI conversations with a mid-market company, the same question surfaces: does our data have to leave the building?

Sometimes it's a compliance officer asking. Sometimes it's a founder who read a data-processing addendum all the way through. Sometimes it's the CFO looking at a per-token invoice that grows every month. The question behind the question is always the same: can we run this ourselves?

The answer in 2026 is yes, for more workloads than most people think, with more caveats than most vendors admit. We run a 120-billion-parameter open-weight model on a single workstation-class GPU box in our own office, and it does real work every day; the box, the throughput, and the kernel flags are written up on our product side (a 96 GB machine holding roughly 48 tokens per second with the full 128K context). This post is what we'd tell a client who asked us whether they should do the same.

One Idea, Three Names

"Private LLM," "self-hosted LLM," and "on-premise LLM" describe the same idea with different emphasis: private stresses whose data it sees, self-hosted stresses who operates it, on-premise stresses where it lives.

The distinction that matters is simpler than the vocabulary: with an API model, your data visits someone else's computer. With a private model, it doesn't.

What You Can Actually Run

The open-weight model ecosystem has matured to the point where the capability tiers are stable and predictable:

Small models (4-12B parameters). Run on a decent consumer GPU or even a modern laptop. Good for classification, extraction, routing, and short drafting: the judgment steps inside most business automations. This tier is underrated because everyone benchmarks it against chat, and its real job is triage.

Mid-size models (20-70B). Need a serious GPU or two, or a high-memory unified-architecture machine. Good general capability: multi-paragraph drafting, summarization of long documents, tool use that mostly works. For many automations, this tier is indistinguishable from the API in output quality.

Large open models (100B+, usually mixture-of-experts). This is what we run. Mixture-of-experts designs mean a 120B model activates only a fraction of its weights per token, so it fits and runs on hardware that would choke on a dense model the same size. Capability lands close enough to commercial APIs that, for bounded automation tasks, the difference stops mattering.

The pattern across all three tiers: the closer your task is to a bounded judgment (read this, classify this, draft this from this data), the smaller the model you can get away with. We've measured that pattern directly: in one bounded submission task on our product testbed, budget-tier models outperformed the premium tiers by a wide margin. Open-ended reasoning is where the frontier API models still earn their fee.

What a Private LLM Really Costs

The hardware math is more honest than people expect:

Entry (roughly $1,500-3,000): one consumer GPU with 16-24 GB of VRAM, runs small models fast and mid-size models at tolerable speed. Enough to power intake and triage automations for a whole department.
Serious (roughly $5,000-12,000): workstation GPUs or a high-memory unified machine, runs mid-size models fast and large MoE models at useful speed. This is the tier our own box lives in.
Rack-grade (from ~$30,000): data-center GPUs, needed only when you have high concurrency or want dense frontier-adjacent models. Most mid-market automation workloads never need this.

Electricity is real but small: a loaded GPU box draws like a space heater, not like a data center. The cost that surprises people isn't hardware or power. It's operations: someone has to own the serving stack, watch it, update models, and know what to do when generation quality drifts after an update. That ownership is the actual price of privacy, and it is not hypothetical: our own box hit a driver watchdog crash at roughly 80,000 tokens of context that took a post-mortem and one kernel flag to fix. Budget for it the way you'd budget for any other piece of production infrastructure, or the box becomes shelfware with a fan.

The Per-Token Crossover

API pricing is friendly at low volume and hostile at high volume. A steady automation that processes a few hundred documents a day can run pleasantly on someone else's meter. An automation that reads every email, classifies every ticket, and drafts every response for a 200-person company crosses into thousands of dollars a month, forever.

Private hardware inverts that curve: high fixed cost, near-zero marginal cost. The crossover arrives faster than most people calculate, because automation workloads are steady by nature. A model that runs the same triage step ten thousand times a day is exactly the workload the fixed-cost box was built for.

When Private Wins

The data cannot leave. Health records, legal matters, financials under NDA, anything a regulator or client contract says stays inside. This is the unbeatable case: no data-processing addendum required, because there is no processor. (If that rule exists on paper but employees are pasting data into public chatbots anyway, you have shadow AI, and a private model is one of the honest fixes.)
The workload is steady and high-volume. The per-token math above.
Latency needs to be boring. A local model's response time doesn't change when a provider has a bad day, deprecates an endpoint, or throttles your tier.
You want the model to hold still. API models change under you: silent updates shift behavior, and a prompt tuned in March behaves differently in June. A model you host changes only when you change it, which is worth a lot in a production automation.

When the API Wins

You need frontier capability. For genuinely hard reasoning, long multi-step agent work, and the newest multimodal features, the commercial frontier models are still ahead. Not by as much as last year, but ahead.
The workload is bursty or experimental. Paying per token is the right structure while you're still discovering whether an automation earns its keep.
Nobody will own the box. An unowned server is worse than an invoice. If there's no one to operate it, rent.

The Answer Is Usually Both

The framing "API versus private" is mostly wrong, because an automation is not one model call. In practice the pattern that works is a mix: a small local model handles the high-volume bounded judgments (classify, route, extract), and a frontier API model gets the calls that genuinely need it. The sensitive fields never leave; the hard reasoning gets the best available brain.

This only works if the automation is built for the job, not for the model: the model behind each step should be swappable without rebuilding the system. That's a design decision made on day one, and it's the one that keeps the private option open even if you start on an API. We wrote about the underlying architecture stance in Why Your AI Needs Rules: deterministic structure around probabilistic parts, whichever machine the probabilistic part runs on.

A Short Diagnostic

If you're weighing the private option, five questions get you most of the way:

Is there a workload where data restrictions currently block AI entirely? That's your private-LLM beachhead.
What's your monthly API spend at target volume, honestly projected? Compare it to a one-time hardware number.
Are your tasks bounded judgments or open-ended reasoning? Bounded favors local; open-ended favors frontier.
Who owns the box? A name, not a committee.
Is your automation built so the model can be swapped? If not, fix that first; it matters more than where the model runs.

We run our own operations this way because it works, not because it's fashionable. If you want a second opinion on your particular mix, talk to us.

The Private LLM Option: What Running AI on Your Own Hardware Actually Takes