Open-source AI guide · updated 2026-11-09
Open-source AI for the computer you already have
The buyer's guide prices the boxes. This page is for the computer already on the desk: which open weights it can hold this week, and which files are larger than the machine.
Four machines
Also on this page: split inference, one model across several machines.
Underclassic's models board is a price list. The buyer's guide is a shelf of machines. This is the third thing people actually ask: the laptop, the gaming PC, the Mac, or the server is already paid for, and a new set of weights landed this week. Which file is worth the disk space?
The recommendations below were checked on November 7, 2026, against that models board (verified November 6) and against the vendor cards linked from it. Where the board is silent and the buyer's guide or a vendor card is the source, the sentence says so. A mixture-of-experts model is quick because only a few billion parameters work on each token. The rest of the file still has to sit in memory. That is the number that decides the tier.
A laptop's 16 GB is shared with the operating system, the browser, and whatever else is open. A graphics card's 16 GB is a separate pool. The two rows below are different machines that happen to wear the same number.
A laptop with about 16 GB of RAM
In the buyer's guide: Mac mini, the 16 GB configuration.
Leave 4 or 5 GB for the machine itself and a 16 GB laptop has about 10 GB for a model and a conversation. That holds a 7 to 12 billion parameter dense model at 4-bit, or something smaller if you want a long document in the same sitting. A 27 billion parameter model belongs in the graphics-card tier. The buyer's guide puts the same ceiling on a 16 GB Mac mini: about 14 billion parameters, comfortably. A Windows laptop with 16 GB of RAM sits in that row; the guide's hardware pick for it is the Mac mini.
Start with Qwen3.5-9B. It is the small sibling of Qwen3.5-397B-A17B, the Qwen row on the models board, and the 9B checkpoint is still the one the local-model catalogs hand a 16 GB machine. Apache 2.0. The 4-bit download is about 6.6 to 7.6 GB, so the operating system keeps a share of the RAM. Ollama's tag is qwen3.5:9b. LM Studio and llama.cpp open the same GGUF. It reads text and images and carries a 256K context window, which you should not try to fill on this laptop.
If you would rather have Google's weights, Gemma 4 12B is the size that belongs here. The models board's Gemma row is the 31B, a desk model whose own 4-bit build is about 17.5 GB and does not fit. The 12B sibling does. The Gemma 4 cards checked this week declare Apache 2.0. Ollama and LM Studio both carry the Gemma 4 family; confirm the 12B tag in the library before a long download, and llama.cpp will load the GGUF either way.
Two models on the board itself are built for this kind of machine. IBM's Granite Switch 4.1 8B is Apache 2.0, about 5 GB at 4-bit, with a 128K context and the Decision and Guardian adapters the board describes, served through llama.cpp. Liquid's LFM3-1.7B, released October 27, fits in about 1 GB at 4-bit with a 256K context, which is the file to run when the laptop also has to be a laptop. Its license is the LFM Open License, not Apache: the published terms cap free commercial use at a revenue threshold and send a larger business to Liquid for a separate agreement. Read that file before you put it in a product. Personal use is the case this page is written for.
For photographs, the board's MiniCPM-V 4.7 keeps most of its experts on disk and asks for about 3 GB of memory. ModelBest's speed figures, 17 tokens a second on an iPad and 13 on an iPhone, are the vendor's, not ours. A 16 GB laptop is more machine than those devices. It is the vision model in this tier. Udgam-384M is also on the board, and the lab's own note says it trails smaller models on neutral tests, so it is not a recommendation.
| Model | License | Memory | Quantization | Setup |
|---|---|---|---|---|
| Qwen3.5-9BAlibaba | Apache 2.0 | About 7 GB at 4-bit (published downloads run 6.6–7.6 GB) | Q4_K_M, or Q5_K_M if you still have RAM free | Ollama (qwen3.5:9b), LM Studio, or llama.cpp |
| Everyday model for this tier. Text and image, 256K context. Sibling of the board's Qwen3.5-397B row. | ||||
| Gemma 4 12BGoogle | Apache 2.0 | A 4-bit file well under 10 GB. The 31B sibling is about 17.5 GB and does not fit | Q4_K_M or Q5_K_M | Ollama or LM Studio (Gemma 4 family), or llama.cpp |
| The board lists Gemma 4 31B. This is the smaller checkpoint from the same April release. | ||||
| Granite Switch 4.1 8BIBM | Apache 2.0 | About 5 GB at 4-bit | Q4_K_M or Q6_K | llama.cpp, which is the path the models board names. Ollama once the tag is current |
| On the models board. One endpoint also serves the Decision and Guardian adapters. 128K context. | ||||
| LFM3-1.7BLiquid AI | LFM Open License v1.0, not Apache | About 1 GB at 4-bit | Q4 or Q8. At this size, Q8 still fits | Ollama, LM Studio, or llama.cpp |
| On the models board. Released October 27. 256K context. Read the revenue clause before commercial use. | ||||
| MiniCPM-V 4.7OpenBMB / ModelBest | Open weights; read the card | About 3 GB. Most experts stay on disk | The published on-device build | The project's own runtime. llama.cpp where a GGUF is posted |
| On the models board. Vision model. Speed figures on the card are the vendor's. | ||||
A gaming PC with a consumer GPU
In the buyer's guide: Oki Home (16 GB RTX 5060 Ti), Ghost Core (24 GB RTX PRO 4000).
An 8 GB card should stay on the laptop list and run those models at a higher quant, entirely on the GPU. Twelve GB is the same advice with a little more room for context. Sixteen GB is the first number that changes the model. Twenty-four GB changes it again. Oki Home is the buyer's guide's 16 GB card, a GeForce RTX 5060 Ti, and it ships a 27B model. Ghost Core is the 24 GB card, an RTX PRO 4000 Blackwell, and it ships Qwen 3.8-27B and Gemma 4 31B. A gaming PC with those amounts of VRAM is the same two rows, whether or not the case says Oki or Ghost on it.
The 27B to download is Qwen 3.8-27B, not the older 3.5 checkpoint. Apache 2.0. Framework, quoted in the buyer's guide, puts the file at 17.6 GB, and Unsloth's hardware table puts a 27B Qwen at about 17 GB in 4-bit and about 14 GB in 3-bit. So a 16 GB card wants the 3-bit file and a short context. A 24 GB card can hold the 4-bit file and still have room to talk. Ghost's preloaded copy is this model.
Gemma 4 31B is the models board's desk-side Google model, Apache 2.0, about 17.5 GB in the 4-bit build Google publishes. That is a 24 GB card. Muse Glimmer 30B is the board's other 24 GB model: Apache 2.0, text and image in, a 128K context, distilled from Muse Spark, and the board says the 4-bit builds fit 24 to 32 GB, a single RTX 5090 being the example. Between Gemma and Glimmer, pick Gemma for a plain chat model and Glimmer if you are handing it pictures.
Nemotron 3.5 Bolt 74B-A9B is the model NVIDIA built for this card. It is on the board. The license on the official GGUF is OpenMDW 1.1. The experts are quantized to about 2 bits so the weights land near 22 GB; Ollama's package, tag nemotron-3.5-bolt, is about 25 GB and needs Ollama 0.40.5 or later (that support landed October 14). The weights fit a 24 GB card. NVIDIA's own fuller recipe, which adds the vision projector and the speculative drafter, is closer to 28 GB, so on a 24 GB card keep the context modest or let a slice spill into system RAM. The published recipe is an NVIDIA build. It is not the Apple Silicon path.
Two more board rows belong in the same drawer. Granite Switch 4.1 also comes in a 30B, Apache 2.0, which at 4-bit is a 24 GB model in the same way Gemma 31B is. LFM2.5-24B-A2B is a 24 billion parameter mixture-of-experts with about 2 billion active, a 131K context, and the board calls it a reasoning agent. A 4-bit file of that size is on the order of 12 to 15 GB, so a 16 GB card can try it. Liquid's speed numbers, 184 tokens a second on an M5 Max and 114 on a Ryzen AI Max+ 395, are the vendor's. The license is Liquid's, and the same revenue clause that applies to LFM3 is the one to read.
| Model | License | Memory | Quantization | Setup |
|---|---|---|---|---|
| Qwen 3.8-27BAlibaba | Apache 2.0 | About 14 GB at 3-bit, about 17 GB at 4-bit. Framework's figure for the file it ships is 17.6 GB | Q3_K_M on a 16 GB card. Q4_K_M or Q5 on a 24 GB card | Ollama, LM Studio, or llama.cpp |
| The 27B the buyer's guide and Ghost are actually shipping. Not a separate row on the models board; the board's Qwen row is the 397B. | ||||
| Arete-27BNous Research | Apache 2.0 | About 17.2 GB at Q4_K_M and about 13.2 GB at Q3_K_M, per bartowski's GGUF | Q3_K_M on a 16 GB card. Q4_K_M or Q5_K_M on a 24 GB card | llama.cpp or LM Studio from a community GGUF; Ollama can pull the same GGUF from Hugging Face |
| Released November 6. A supervised fine-tune of Qwen 3.8-27B that Nous says removes state-framed answers on politically sensitive topics. The before-and-after scores on the card are Nous's own. | ||||
| Gemma 4 31BGoogle | Apache 2.0 | About 17.5 GB at 4-bit, Google's figure | Q4_K_M on 24 GB. Q6 if you have 32 GB | Ollama (gemma4:31b), LM Studio, or llama.cpp |
| On the models board as the desk-side Gemma. Ghost preloads it. A 16 GB card does not hold it. | ||||
| Muse Glimmer 30BMeta | Apache 2.0 | 4-bit builds fit 24–32 GB, per the models board | Q4_K_M | Ollama, LM Studio, or llama.cpp |
| On the models board. Text and image, 128K context, distilled from Muse Spark. The board's example card is an RTX 5090. | ||||
| Nemotron 3.5 Bolt 74B-A9BNVIDIA | OpenMDW 1.1 | About 22 GB for the official GGUF. Ollama's package is about 25 GB. The vision-plus-drafter recipe is closer to 28 GB | The native 2-bit (IQ2_XXS) checkpoint. Do not requantize it to 4-bit and expect it to shrink | Ollama (nemotron-3.5-bolt, version 0.40.5 or later) or llama.cpp. LM Studio once it has this architecture |
| On the models board. Built for RTX and DGX Spark. NVIDIA path, not Apple Silicon. | ||||
| LFM2.5-24B-A2BLiquid AI | LFM Open License, not Apache | On the order of 12–15 GB at 4-bit, from the parameter count | Q4_K_M | Ollama, LM Studio, or llama.cpp |
| On the models board. 24B total, about 2B active, 131K context. Speed figures on the card are the vendor's. | ||||
A high-memory Apple Silicon Mac
In the buyer's guide: Mac Studio, Mac mini, the 64 GB M5 Pro.
Apple's machines are the odd case where the graphics memory and the system memory are the same pool, and the bandwidth is what makes a reply feel quick. The buyer's guide's Mac Studio runs from 36 GB to 256 GB. The Mac mini in an M5 Pro configuration reaches 64 GB, which the guide scores as a 30B-class machine. MLX and llama.cpp both cover inference. Most fine-tuning code still assumes NVIDIA, which is the caveat Tom's Hardware and the buyer's guide already make.
From 36 to 48 GB, stay with the gaming-PC models and give them a higher quant: Qwen 3.8-27B and Gemma 4 31B at Q5 or Q6, Muse Glimmer 30B at Q5. Recursive Flint 40B-A2.5B is the models-board model written for this bracket. Apache 2.0, about 22.6 GB at INT4, a 262K context, built as a coding agent. Recursive's figure of 125 tokens a second on an M4 Pro is the vendor's. A 36 GB Mac runs it with little left over. A 48 GB Mac runs it properly.
At 64 GB the interesting new file is Ripple-65B Preview, the Apache 2.0 dense model Ricursive published on November 4 and that Issue 97 led with. It is not a row on the models board yet. The BF16 weights are about 130 GB. A community Q4 GGUF is about 39 to 40 GB, and a Q5 is about 48 GB. On a 64 GB Mac the Q4 file is a capacity target: it loads, and a long context will push you into swap. llama.cpp merged the architecture on November 5, build 12167. Ollama and LM Studio load the GGUF once they ship that build, which as of this writing is not a promise. Ricursive's own NVFP4 checkpoint is 57 GB because only the MLP layers are 4-bit and the attention stays fuller, and the company points that file at a DGX Spark, not at a 64 GB Mac.
At 128 GB the models board finally has peers for the file the buyer's guide uses as its yardstick. That yardstick is gpt-oss-120b in MXFP4, about 63 GB, the model the guide says a 128 GB GMKtec holds. It is not a row on the November 6 board. The board's models in the same band are Mistral Small 4 and Llama 4 Scout. Mistral Small 4 is about 119 billion parameters with about 6 billion active, Apache 2.0, and a Q4_K_M GGUF is about 68 to 73 GB, so a 96 GB Mac is the minimum and 128 GB is where it stops feeling crowded. A Q3 file near 54 GB is the 64 GB stretch, and it is a stretch. Llama 4 Scout is about 109 billion total and 17 billion active, under Meta's community license, with the long-context role the board gives it. Qwen 3.8 Flash-Next, about 125 billion total and 6 billion active, is the model owners are already posting about on a 128 GB M5 Max and on the 128 GB EVO-X2. It is under the Qwen community license, which is a commercial license with a user-count clause, not Apache. A 4-bit file of that size is on the order of 70 GB.
At 192 and 256 GB the board's open flagships become single-machine models, some of them barely. Qwen3.5-397B-A17B is Apache 2.0, 397 billion parameters with 17 billion active. Unsloth's table puts the 3-bit file near 180 GB, which is a 192 GB machine, and the 4-bit file near 214 GB, which is a 256 GB Ultra and may still need macOS persuaded to hand the GPU more than its default share of unified memory. Llama 4 Maverick is the same class, about 400 billion total and 17 billion active, community license, and the board notes EU multimodal limits. DeepSeek V4.1 Flash is the model Framework names for a 192 GB desktop at Q2. The board lists it at about 284 billion parameters, about 13 billion active, and says the weights behind the current Flash price are the October 17 checkpoint. Muse Spark Open, 816 billion total and 42 billion active under the Muse Model License, is the model the board says runs on two M5 Ultra Mac Studios. One Ultra, however large, is not that configuration.
| Model | License | Memory | Quantization | Setup |
|---|---|---|---|---|
| Recursive Flint 40B-A2.5BRecursive | Apache 2.0 | About 22.6 GB at INT4 | The published INT4 build | Ollama, LM Studio, or llama.cpp on Apple Silicon |
| On the models board. Coding agent, 262K context. Comfortable from 48 GB. The 125 tok/s figure is Recursive's, on an M4 Pro. | ||||
| Ripple-65B PreviewRicursive Intelligence | Apache 2.0 | BF16 about 130 GB. Q4 GGUF about 39–40 GB. Q5 about 48 GB. NVFP4 about 57 GB | Q4_K_M on a 64 GB Mac if you accept a short context. Q5 on 128 GB. NVFP4 on a Spark | llama.cpp build 12167 or later. Ollama and LM Studio after they ship that build |
| Issue 97, not yet a models-board row. Dense 65B, 1,048,576 context on the card. One million tokens of cache is a different purchase. | ||||
| Mistral Small 4Mistral | Apache 2.0 | Q4_K_M about 68–73 GB. Q3 about 54 GB. Q5 about 80–85 GB | Q4_K_M on 128 GB. Q5 if you have it. Q3 only as a 64 GB experiment | Ollama from the Hugging Face GGUF, LM Studio, or llama.cpp |
| On the models board. About 119B total, about 6B active, so it answers faster than the file size suggests. | ||||
| Llama 4 ScoutMeta | Llama community license | Same band as Mistral Small 4: a 4-bit file on the order of 60–70 GB | Q4_K_M | Ollama, LM Studio, or llama.cpp |
| On the models board. About 109B total, about 17B active. The board's long-context open path. Read the community license before you ship a product on it. | ||||
| Qwen3.5-397B-A17BAlibaba | Apache 2.0 | About 180 GB at 3-bit, about 214 GB at 4-bit | 3-bit on 192 GB. 4-bit on a 256 GB Ultra | llama.cpp or MLX. Ollama and LM Studio if the split GGUF is more than they want to manage |
| The Qwen row on the models board. 397B total, 17B active. macOS may need its GPU memory limit raised before a 256 GB machine will hold the 4-bit file. | ||||
| Llama 4 MaverickMeta | Llama community license | Same class as the 397B: a 256 GB Ultra at a low quant | 3-bit or 4-bit, and only after you have measured the file | llama.cpp or MLX |
| On the models board. About 400B total, about 17B active. The board flags EU multimodal limits. | ||||
A home server, or more than one GPU
In the buyer's guide: NVIDIA DGX Spark, GB10 partner boxes, Surface RTX Spark, GMKtec EVO-X2, AMD Ryzen AI Halo, 192 GB Ryzen AI Max+ boxes, Framework Desktop, tinybox, split-inference hardware.
A 128 GB home server holds the same weights as a 128 GB Mac. Mistral Small 4 at Q4, Llama 4 Scout, Qwen 3.8 Flash-Next, and Ripple-65B at Q4 or Q5 are the files, and the Mac section above is the memory table. What changes is the software. CUDA and vLLM are the native path on the NVIDIA boxes. The AMD Strix Halo machines, the GMKtec EVO-X2, the Ryzen AI Halo, and the 192 GB PRO 495 boxes, get there through llama.cpp, Vulkan, or ROCm. Framework Desktop is the same AMD silicon in a case you can cluster.
On NVIDIA the extra file is the one the company aimed at the Spark. Nemotron 3.5 Bolt, already in the gaming section, is easy on a 64 GB or 128 GB DGX Spark or on the GB10 partner boxes and the Surface RTX Spark, because the 24 GB recipe has memory to spare and the context can grow. Ripple's NVFP4 checkpoint, 57 GB, is the file Ricursive says fits a Spark. Nemotron 3.5 Super VL is the board's 120B vision-language model, OpenMDW 1.1. A 4-bit file of a 120B model is on the order of 70 GB, so it wants the 128 GB machines, or a split across more than one smaller GPU.
At 192 GB, Framework's own pick, recorded in the buyer's guide, is DeepSeek V4.1 Flash at Q2. The models board lists that model at about 284 billion parameters and about 13 billion active, and notes that the weights behind today's Flash price are the October 17 checkpoint. Qwen3.5-397B-A17B at 3-bit, about 180 GB, is the other file a 192 GB box can attempt. Both are tight. Leave room for the system.
tinybox is the multi-GPU case: 64 GB of VRAM across four Radeon cards in the base build, up to 192 GB with six. The buyer's guide's estimate for the four-card machine is a 70B-class model split across the GPUs. That is Gemma-and-Qwen territory spread out, and it is also where Nemotron Super's 4-bit file can land if a single card cannot hold it. The six-card, 192 GB build reaches the same Q2 and Q3 mixture-of-experts files as the big unified-memory boxes. Splitting a model is llama.cpp or vLLM work. It is not an Ollama one-liner, and the machine draws 1,700 watts.
Two machines is a different tier again, and the board is specific. GLM-5.3-Flash has open weights on Hugging Face. The board leaves the parameter count to the card and records a speed claim, 36 to 92 tokens a second on two DGX Sparks, as self-reported by Mia's AI Lab and not verified by Underclassic. Muse Spark Open is the two-Mac case: two M5 Ultra Studios, the Muse Model License, 816 billion parameters with 42 billion active. A single home server does not become that by wishing.
Three models on the board are larger than a home, even a well-built one. DeepSeek V4 Pro is about 1.6 trillion parameters with about 49 billion active. Mistral Large 4 is about 1.05 trillion with 52 billion active, under Mistral's ecosystem open-weight license rather than Apache. Kimi K3.1 Pro, published in November, is about 2.8 trillion with 104 billion active, under Moonshot's own license. They are real open weights. They are also a cluster, or someone else's cluster. Downloading them onto a desktop does not make them fit.
| Model | License | Memory | Quantization | Setup |
|---|---|---|---|---|
| Mistral Small 4, Llama 4 Scout, Ripple-65BSee the Mac section | Apache 2.0, Llama community, Apache 2.0 | The 128 GB band: roughly 40–75 GB depending on the file | Q4 or Q5, as in the Mac table | llama.cpp on AMD and NVIDIA. vLLM on NVIDIA. Ollama where the tag exists |
| Same weights as a 128 GB Mac. The server section does not pretend they are different models. | ||||
| Ripple-65B NVFP4Ricursive Intelligence | Apache 2.0 | About 57 GB. Attention stays at higher precision, so it is larger than the Q4 GGUF | The official NVFP4 checkpoint | The Spark path Ricursive documents. llama.cpp for the GGUF |
| The file the company points at a DGX Spark. | ||||
| Nemotron 3.5 Super VLNVIDIA | OpenMDW 1.1 | A 4-bit file of a 120B model is on the order of 70 GB | Q4_K_M on 128 GB unified memory, or split across GPUs | llama.cpp or vLLM. Ollama and NIM where the tag is current |
| On the models board. Vision and language. Bolt, above, is the 24 GB cut of this line. | ||||
| DeepSeek V4.1 FlashDeepSeek | MIT on the V4 weight cards | Framework runs the Q2 file on 192 GB | Q2 on a 192 GB machine, which is Framework's own pick | llama.cpp. Confirm the architecture build before you assume Ollama has it |
| On the models board at about 284B total and about 13B active. The board says today's Flash price serves the October 17 checkpoint. | ||||
| GLM-5.3-FlashZ.ai | Open weights; the board points at the Hugging Face card | Two DGX Sparks in the only figure the board records | The EXL3 build in that two-Spark report | The two-Spark stack. Not a single-card download |
| On the models board. The 36–92 tok/s range is Mia's AI Lab, self-reported, and the board says Underclassic has not verified it. | ||||
| Muse Spark OpenMeta | Muse Model License, limited commercial terms | Two M5 Ultra Mac Studios, per the models board | The build Meta describes for that pair of machines | MLX, which is the path the board names |
| On the models board. 816B total, 42B active, 1M context. One home server is the wrong shape. | ||||
Split inference: one model, several machines
In the buyer's guide: hardware that links up for split inference.
Underclassic has spent nearly a hundred issues scoring a fight between two ideas of where AI lives: on the machine you own, or in a cluster someone else owns. Split inference is the awkward, promising third answer. Instead of asking which computer runs the model, it asks which part of the model each computer should run.
The trick works because a language model is a stack of layers, and every answer happens in two phases. First the model reads your whole prompt at once. That is prefill, and it wants raw compute. Then it writes the reply one token at a time. That is decode, and it mostly wants fast memory. Either seam, between the layers or between the phases, can be cut, and people are cutting it in three places.
1. Your device and the cloud share one model Mostly research
The oldest version. The first layers run on your phone or laptop, and what leaves the device is an intermediate result rather than your raw words, which a server then finishes. Researchers have called this split computing since about 2017, and a 2021 survey maps the field. That intermediate result is harder to read than plain text, though not impossible, so the privacy gain is real but partial.
For large language models it is still mostly research. What phones and laptops ship today is routing, not splitting: Apple Intelligence runs a small model on the device and sends harder requests, whole, to larger models on Apple's Private Cloud Compute. We could not find a consumer product that splits one language model's layers between your device and a data center.
2. Several machines you own act as one Shipping, for hobbyists
This is the version you can install tonight. exo links Macs and Linux machines into a single cluster, finds the other devices on its own and decides how to shard the model across them. On Macs with Thunderbolt 5 running macOS 26.2 or later it talks over the cable with RDMA, and its README shows four 512 GB Mac Studios serving DeepSeek v3.1 and Kimi K2 Thinking together. llama.cpp's RPC backend lets one machine borrow the GPUs of others over the network, and its own documentation calls it a fragile, insecure proof of concept that should never face an open network. Distributed Llama does the same job with tensor parallelism over Ethernet, all the way down to Raspberry Pis.
The models board already assumes this world. Meta's Muse Spark Open is listed for two M5 Ultra Mac Studios, and the only GLM-5.3-Flash speed figure we carry comes from two DGX Sparks. The catch is the wire. Pieces of the model have to talk on every token, so a cluster is only as quick as its slowest link, and two boxes are rarely twice as fast as one. exo says its tensor parallelism reaches up to 1.8 times the speed on two devices and 3.2 times on four; those are the project's own numbers.
3. The data center splits the reading from the writing In production
The third kind happens far from your desk. Prefill/decode disaggregation runs the prompt-reading phase on one pool of chips, hands the model's working memory, the KV cache, across a fast network, and lets a second pool write the reply. NVIDIA's Dynamo is built around the idea, vLLM offers it as an experimental feature, SGLang and llm-d ship it, and Mooncake is the disaggregated platform behind Moonshot's Kimi.
It was the quiet detail in Issue 99: NVIDIA GPUs doing prefill and Rebellions R200 chips doing decode inside the same NVIDIA rack design, in racks due in 2028. Once the two phases live on different chips, nothing in principle says they have to live in the same building.
Why it is the bridge
Taken together, the three cuts blur the line Underclassic draws every day. The second turns a few owned computers into a small private cluster. The third shows that the giant clusters no longer treat a model as one indivisible job. The first, if it matures, would let the private part of a request stay on your device while the heavy part rents a data center.
None of that settles the axis, and today the only kind most readers can use is the one where they buy more computers. But it is the clearest sign yet that local and central are not two separate futures. More and more, they are two halves of the same model. The hardware for the second kind, with prices, is in the buyer's guide.
Tools and services that do split inference
On machines you own
- exo. Apache 2.0. Clusters Macs and Linux machines on Apple's MLX, with RDMA over Thunderbolt 5 on macOS 26.2 or later.
- llama.cpp RPC. MIT. One llama.cpp host offloads work to GPUs on other machines; the project calls it a proof of concept and says never to expose it to an open network.
- Distributed Llama. MIT. Tensor parallelism across home devices over Ethernet, from Raspberry Pis to GPU machines.
- MLX distributed. The multi-machine layer of Apple's MLX framework, which exo builds on.
- Petals. MIT. Runs a model BitTorrent-style across volunteers' GPUs in a public swarm, or in a private swarm you set up. Strangers process your prompts in the public swarm, and the code has not been updated since September 2024.
In the data center: prefill/decode
- NVIDIA Dynamo. Open-source orchestration layer over vLLM, SGLang and TensorRT-LLM that splits prefill and decode into separately scaled GPU pools.
- vLLM disaggregated prefill. Built into vLLM and marked experimental in its own docs.
- SGLang PD disaggregation. SGLang's documented prefill/decode split.
- llm-d. Apache 2.0 and Kubernetes-native. Reached v1.0 in October 2026 and runs on vLLM and SGLang.
- Mooncake. Apache 2.0. A KV-cache-centric disaggregated design and the serving platform for Moonshot's Kimi.
Commercial services
- AWS, with llm-d. AWS's March 16, 2026 post brings disaggregated inference to SageMaker HyperPod and EKS with an llm-d container tuned for its EFA network. You run it on your own AWS capacity; it is not a per-token API.
- Splitting one model between your device and the cloud: no commercial service we could verify. Apple's Private Cloud Compute is the nearest thing, and it routes whole requests rather than splitting a model.
- Pooling machines you own: no paid service is needed or offered that we could verify. The tools above are free and open source.
Every link checked November 9, 2026.
The pattern, if you only remember one, is dull and reliable. Match the file size to the memory you have, then leave a margin for the context and the operating system. A newer model with a smaller active-parameter count is not smaller on disk. When a license is Apache 2.0, the page says so. When it is a community license, an OpenMDW grant, or a revenue cap, the page says that too, because "open" on the models board covers all of them and they are not the same permission.
Weights move every week. Edit guides/open-source-ai/models.json and run the builder that writes this page. The machines those files run on, with prices, are the buyer's guide. The API prices, for the days you do not want a fan running, are the models board.
Sources: Underclassic models board, verified November 6, 2026; Underclassic buyer's guide, prices as of November 2, 2026; Ripple-65B Preview model card; Nemotron 3.5 Bolt 24 GB GGUF; Unsloth, Qwen3.5 local hardware table; exo README; llama.cpp RPC README; NVIDIA Dynamo README; AWS, disaggregated inference with llm-d; Arete-27B model card; bartowski, Arete-27B GGUF sizes.