An SRE's guide to deploying Large Language Models, Part 3: the hardware and software landscape
Part 1 and Part 2 were about the model: what it is, how it is trained into an assistant, and the mechanics of serving it. This part is about the metal underneath, the chips you actually run it on, and the software that drives them. It is the part most likely to read like a spec sheet, so the whole thing is built around a single trick for making spec sheets legible, and that trick comes straight out of where part 2 left off.
What actually matters in the hardware
Part 2 ended on two facts about serving that quietly decide everything about the hardware you buy. Prefill, the pass that reads your whole prompt at once, is compute-bound: it is a big pile of matrix multiplications happening together, limited by how fast the chip can multiply. Decode, the loop that produces one token at a time, is bound by memory speed: each token has to stream a large fraction of the model’s weights out of memory and through the compute exactly once, and it is limited not by how fast the chip multiplies but by how fast it can read those weights out.
Hold onto that split, because it is the whole reason a GPU spec sheet looks more intimidating than it is. Strip away the marketing and only three numbers decide whether a given chip can serve a given model, and how fast:
- Capacity. How much memory sits wired directly to the multiply units. The model’s weights, plus its KV cache, plus a little working room, all have to fit in there at once. If they do not fit, nothing else matters: you either cannot run the model, or you spill onto a second chip and start paying the interconnect tax.
- Compute. How fast the chip does the matrix multiplications, which is what governs prefill. Most of that speed comes from dedicated hardware built for exactly this, and from whether that hardware can multiply your quantized weights directly rather than converting them back up first.
- Bandwidth. How fast weights move from memory into the multiply units, which is what governs decode. This is the number that sets your tokens per second for a single stream.
That is the entire lens. Capacity decides if the model fits, compute decides how fast prefill and the format tricks run, bandwidth decides how fast decode runs. There is a fourth thing that keeps coming up, interconnect, the speed at which chips talk to each other, but it is not really a separate lever. It is how you buy more capacity, compute, and bandwidth than a single chip has, by ganging several together, so it shows up inside each section below rather than as its own.
Every vendor in this part, NVIDIA, Apple, AMD, AWS, and Google, is just a different set of answers to those three numbers, plus a different story for stitching chips together. Once you can read a chip as “this much capacity, this fast at these formats, this much bandwidth,” the rest of the spec sheet is noise.
The metal: chips you buy
NVIDIA, Apple, and AMD make silicon you can own outright, a chip or a card that goes in a machine you control. Everything the cloud rents you later is one of these underneath, so the metal comes first.
NVIDIA
NVIDIA started out making graphics chips, and a graphics chip is already a machine for doing enormous numbers of small multiplications in parallel, which turns out to be exactly the shape of a transformer. So as this workload grew, NVIDIA leaned into it, and the clearest expression of that is the tensor core: a block of silicon that does nothing but multiply small matrices, the operation from part 1, as fast as it can.
NVIDIA chips come in architecture generations, and each new generation packs in more and faster tensor cores and teaches them to multiply more number formats natively. That last part is the whole compute story, because a format the tensor cores handle directly runs at full speed while one they do not has to be converted up first. The current generation is Blackwell, whose tensor cores go all the way down to NVFP4, NVIDIA’s own optimized four-bit floating-point format.
| Generation | Datacenter | Consumer / workstation | Native low-precision formats |
|---|---|---|---|
| Ampere (2020) | A100 | A10G, RTX 30 | INT8, INT4 |
| Ada Lovelace (2022) | L4, L40S | RTX 40 | INT8, FP8 |
| Hopper (2022) | H100, H200 | (datacenter only) | INT8, FP8 |
| Blackwell (2024) | B200, B300 | RTX 50, RTX PRO 6000 | INT8, FP8, FP6, FP4 |
The line to read off it: FP8 arrives with Hopper and Ada, FP4 with Blackwell. So the same FP8 model that runs at full speed on an H100 only saves memory on an older A100, and an FP4 model that flies on a Blackwell card has nothing to run on natively before it.
Compute
Within a generation the tensor-core math is identical on the cheap silicon and the expensive silicon: a workstation RTX PRO 6000 runs the exact same native FP4 matmul as a datacenter B200. So the compute lever, on NVIDIA, comes down to one question, does the card’s generation natively multiply the format you plan to quantize to. If it does, quantization is a speed win; if it does not, it is only a memory win. What the datacenter premium actually buys is not the math, it is the memory and the interconnect around it.
Memory
Two kinds of memory split the lineup. The datacenter chips use HBM, stacked right next to the die and wired over a very wide bus: fast, roomy, and expensive, giving an H100 80 GB, an H200 141, a B200 180, and the Blackwell Ultra 288. The consumer and workstation cards use GDDR soldered around the board: far cheaper, and less of both capacity and speed, which is why the flagship workstation card tops out at 96 GB and a gaming 5090 at 32. HBM is the clearest single marker that a chip is a datacenter part, and it shows up in both of the levers it touches, capacity and bandwidth.
Bandwidth, the decode lever, spans a wide range. Consumer memory runs from the A10G’s 600 GB/s through the L40S’s 864 to GDDR7’s 1.8 TB/s on the newest workstation cards. Datacenter HBM starts where that ends: 3.35 TB/s on an H100, 4.8 on an H200, around 8 on a B200. Even the fastest workstation card moves weights roughly four and a half times slower than a B200, and since decode speed rides this number almost entirely, that is about the per-stream token-rate gap between the two.
Unified memory
Everything above is dedicated memory: the GPU has its own HBM or GDDR, kept separate from the CPU’s system RAM. NVIDIA sells the other kind too, in DGX Spark, a small Grace Blackwell box where the CPU and GPU share one 128 GB pool. That buys huge capacity for the size, enough to hold a two-hundred-billion-parameter model, but the pool is ordinary LPDDR at 273 GB/s, fifteen to thirty times slower than datacenter HBM. So it fits big models and decodes them slowly. You can cluster two Sparks over their built-in ConnectX-7 link to reach 256 GB, but that link is 200-gigabit Ethernet, a fraction of NVLink, so it stays a development-and-prototyping box rather than a serving cluster. Hold that capacity-versus-bandwidth shape, because it repeats on every unified-memory machine in the sections below.
Interconnect
When a model is too big to fit into the VRAM of a single GPU, it has to be split across several GPUs. This is called tensor parallelism. Rather than handing each GPU whole layers, you give every GPU a slice of every layer’s weight matrices, so inference settles into a two-step rhythm. For each block, every GPU runs its own slice on the incoming token and produces a partial result, its share of the answer computed from its shard of the weights. Then the GPUs exchange those partials and add them together, an operation called an all-reduce, so each one ends up holding the complete output of the block, ready to feed the next. The catch is that this exchange happens after every single block, twice in every layer, and it runs at the speed of the wire between the GPUs. That is why the interconnect, and not the compute, is what makes or breaks splitting a model across chips.
NVIDIA’s fast wire is NVLink, a direct GPU-to-GPU link that has climbed from 600 GB/s per GPU on the A100 to 900 on the H100 to 1.8 TB/s on Blackwell. On top of it, NVSwitch is a switch that lets every GPU in a box reach every other at that full rate no matter how many there are, which is how seventy-two Blackwell GPUs get wired into a single rack that behaves like one enormous GPU. Without NVLink, chips fall back to PCIe, the ordinary bus to the CPU and roughly fourteen times slower, on which the after-every-block exchange spends more time talking than computing and tensor parallelism stops paying off. And here the tier split returns: the cheap cards, the L40S and the RTX PRO 6000, are PCIe-only, so they are excellent single-chip and poor at pooling.
So NVIDIA’s answer is the full ladder on all three levers, backed by an interconnect nobody else matches, which is why it is the default. The catch worth keeping straight is that tier split: the cheap cards keep the tensor-core math, the native FP8 and FP4 that make quantization fast, but trade HBM for GDDR and NVLink for PCIe. You keep the speed and lose the ability to pool many chips.
Apple Silicon
Apple is the odd one out, and the most interesting answer to the three levers, because it wins the first one, capacity, in a way no datacenter GPU can touch, and loses the third, bandwidth, in a way no datacenter GPU would tolerate.
The way it gets there is unified memory, the same idea as DGX Spark a moment ago: one pool of memory shared by everything on the chip, with no separate VRAM. Apple doubles down on it. Where most of the industry treats shared memory as a compromise, Apple builds the whole chip around it and wires a very wide bus to the pool, so the shared memory is not only large but reasonably fast. A Mac Studio configured with 512 GB of it can hold a 671-billion-parameter mixture-of-experts model at 4 bits, the kind that otherwise needs a multi-GPU datacenter node, on a machine that sits on your desk and draws about a kilowatt. Nothing else gives you that much model-capacity per dollar. The ladder runs from 32 GB on a base chip up through 128 GB on a Max and 512 GB on an Ultra, and that capacity costs a fraction of the equivalent HBM.
Three kinds of compute share that pool, and it is worth naming them because Apple’s branding makes them easy to confuse. There is the CPU; there is the GPU; and there is the Neural Engine, a separate block of sixteen cores that Apple built for running its own on-device models, the vision and Apple-Intelligence features. For a large language model the one that matters is the GPU, driven through Metal. The Neural Engine sounds like it should be the AI part, but it is tuned for Apple’s fixed workloads and is awkward to aim at a general transformer, so the local inference tools do not touch it.
For a long time that GPU was the weak spot: it could do the matmuls but had no dedicated matrix hardware, so it unpacked quantized weights back to 16-bit and multiplied them the slow way. The M5 fixed it, with the second thing that gets confused with the Neural Engine. Every GPU core now carries a Neural Accelerator, which despite the name is a completely different unit from the Neural Engine: a small matrix-multiply block, a tensor core, built inside the GPU itself. So the Neural Engine is a separate island Apple mostly keeps for its own features, while the Neural Accelerator is the tensor core in the GPU that finally makes Apple competitive at the operation a language model is made of. With it, the GPU multiplies 16-bit and 8-bit natively, which Apple markets as more than four times the AI throughput of the M4; 4-bit is still a storage format it unpacks before the math. Compute, in any case, is rarely what limits you here.
What limits you is bandwidth, and this is where the desk machine meets its ceiling. Apple’s memory bandwidth climbs from around 120 GB/s on a base chip to 273 on a Pro, roughly 550 on a Max, and 1.2 TB/s on the current M5 Ultra, which sounds enormous until you put it beside a datacenter part. A single H200 moves weights at 4.8 TB/s, a B200 at nearly 8, and a datacenter node bolts eight of them together. So even Apple’s flagship decodes a given dense model several times slower per stream than one datacenter GPU, and cannot come near a node’s aggregate. This is exactly why the mixture-of-experts models from earlier suit a Mac so well: a model that is huge in total but activates only a few billion parameters per token asks little of the bandwidth while filling all that capacity, the one shape of model where Apple’s strengths and weaknesses line up in your favor.
On software, the thing to know is MLX, Apple’s own array framework built from the ground up around unified memory. Because everything lives in one pool, MLX never copies tensors between a host and a device; the model loads once and every engine reads the same buffers. That, plus aggressive fusing of operations to avoid touching memory more than necessary, makes it meaningfully faster than llama.cpp’s Metal path for generation, and it has become the serious way to run models locally on a Mac, to the point that Ollama switched its Apple backend to it. The cost is that MLX is Apple-only; it is worthless the moment your inference moves to a Linux box with an NVIDIA card.
Finally, scaling, where Apple essentially bows out of the datacenter conversation. Within a package it has one clever move, UltraFusion, which bridges two Max dies across a very wide, very fast link so software sees a single GPU and a single pool. That is how an Ultra is built, and it stops at two dies. Beyond one machine there is no NVLink and no InfiniBand; you cluster Macs over Thunderbolt, now with a remote-memory mode, and frameworks like EXO will shard a model across four Mac Studios to run something trillion-parameter-sized at reading speed for well under the price of one datacenter GPU. But a Thunderbolt link is on the order of a hundred times slower than NVLink, which starves the constant all-reduce that tensor parallelism needs, so this works for running one enormous model slowly at home and falls apart as a way to serve thousands of concurrent requests.
In the three-lever frame: Apple wins capacity per dollar outright, brings genuinely respectable and now matrix-accelerated compute, but is bandwidth-limited on decode and has no real multi-node story. The best tool going for running a big model on your own hardware, and the wrong tool for high-throughput serving.
AMD
AMD is the one company with a genuine claim to being the alternative to NVIDIA, and its whole pitch reads straight off the three levers: it wins the capacity one, it has drawn level on the other two, and the reasons you used to hear for not buying it, the software and the way its chips talk to each other, have mostly gone away.
Start with capacity, because that is the real argument. Generation for generation, an AMD Instinct card carries markedly more high-bandwidth memory than the NVIDIA part it competes with. The MI300X shipped with 192 GB against the H100’s 80 and the H200’s 141; the MI325X pushed that to 256 GB; the current MI350X and MI355X carry 288 GB, which matches NVIDIA’s biggest Blackwell and beats the mainstream B200. More memory per chip is not a vanity number. It means the model plus its KV cache fits on fewer GPUs, which means less tensor-parallel sharding, which means you dodge the interconnect tax and can run bigger batches and longer contexts on a single node. A single MI350X holds a 405-billion-parameter model in FP8 by itself. When people say AMD wins the “fit the whole model on one chip” comparison even when the raw compute is close, this is what they mean.
On compute and bandwidth the two are now roughly even per chip. Bandwidth was never the problem: the MI300X already moved weights at 5.3 TB/s, the MI325X at 6, and the CDNA 4 cards at 8, right in line with or ahead of the equivalent NVIDIA HBM. Compute took one more step to catch up. The first Instinct generation, the MI300X and MI325X, tops out at native FP8; it has no FP4 or FP6 circuitry, so the aggressive four-bit path that Blackwell opened simply is not there, and FP8 is your floor. The CDNA 4 chips, the MI350 series, fixed that by adding native FP4 and FP6 matrix units, AMD’s direct answer to Blackwell’s tensor cores and, on FP6 specifically, actually faster. So the precision story is a generation behind NVIDIA’s, but it has arrived.
The honest weak spot is interconnect, and it is worth being precise about why. NVIDIA wires the GPUs in a server through NVSwitch, a switched fabric where every GPU talks to every other at full speed no matter how many there are, and it now stretches that to a coherent 72-GPU domain. AMD wires its eight GPUs in a direct all-to-all mesh instead, with no switch in the middle, so the collective bandwidth that tensor parallelism leans on thins as you add GPUs, and the whole scale-up domain stops at eight. For one big model spread across many chips, that gap is real and measurable. AMD’s fix, a rack-scale 72-GPU system on the coming MI400 generation, lands in the second half of 2026 and finally answers NVIDIA’s rack, though in its first year it runs over a repurposed Ethernet fabric rather than the open standard AMD is nominally building toward. Until that ships, “spread a huge model across a whole rack” is the one thing NVIDIA still does that AMD cannot match.
The last piece is software, and this is the part that genuinely changed. For years the reason not to buy AMD was ROCm, its answer to CUDA, which lagged badly enough that you would fight the tooling for weeks. As of 2026 that is largely over for inference specifically: both vLLM and SGLang treat AMD as a first-class target, AMD ships supported images and benchmarks, and independent testing puts a well-configured MI300-class card at something like ninety to ninety-five percent of an H100’s inference throughput. The gaps that remain are real but narrow: there is no AMD equivalent of TensorRT-LLM, the hand-tuned FlashAttention-3 kernels are CUDA-only, and the training story still trails. For serving, though, the software is no longer the reason to say no. And since AMD is the cheapest way to rent a high-memory GPU in the cloud, the capacity advantage comes with a price advantage attached.
There is a local angle too, and it rhymes exactly with Apple’s. AMD’s Ryzen AI Max chip, “Strix Halo,” puts up to 128 GB of unified LPDDR5X memory behind an integrated GPU, so a mini-PC can load a 120-billion-parameter model and chat with it, bounded, as always, by the modest bandwidth of that shared pool rather than by compute. It is the same capacity-wins-locally, bandwidth-caps-you bargain as a Mac, aimed squarely at NVIDIA’s little DGX Spark box, and usually cheaper.
In the three-lever frame: AMD wins capacity, has pulled level on compute and bandwidth, and has quietly fixed the software for inference. Hold one caveat in reserve, that its multi-GPU interconnect trails NVIDIA’s until the MI400 rack ships, and it is a fully credible place to serve models, often for less money.
The cloud: chips you rent
AWS and Google you cannot own, only rent by the hour. AWS is largely a repackager, wrapping the metal above into instances and adding a custom chip of its own; Google builds its own TPU and lets you use it only inside Google Cloud. The theme here is less new silicon than how you get your hands on it.
AWS
AWS is a different kind of entry on this list, because it does not really make chips, it rents them, and it is the one place that rents you everything above: the whole NVIDIA ladder, packaged into instances, plus a pair of its own custom accelerators. So the AWS section is less about new silicon and more about how the silicon you have already met gets sold to you by the hour, and the single most useful thing to carry into it is the meaning of the two instance letters.
A GPU instance whose name starts with g is the cost-optimized tier. Its GPUs are the consumer-and-workstation lineage from the NVIDIA section, the ones with GDDR memory instead of HBM and, crucially, no NVLink, wired to each other only over PCIe. A p instance is the datacenter tier: HBM GPUs, NVSwitch inside the box, and the fast cross-node networking that frontier serving needs. That one letter tells you which half of everything the NVIDIA section said you are getting, all three levers at once. The g-ladder runs g5 (the Ampere A10G, 24 GB, no FP8), through g6 and g6e (the Ada L4 and L40S, which add native FP8, at 24 and 48 GB), to the current standout g7e, a Blackwell RTX PRO 6000 with 96 GB of GDDR7, native FP4, and 1.6 TB/s of bandwidth, a genuinely capable single-GPU inference box. The p-ladder runs p4 (A100), p5 and p5e (H100 and H200), and p6 (the B200 and Blackwell Ultra B300), and tops out in the UltraServers, which wire seventy-two Grace-Blackwell GPUs into a single NVLink domain with more than thirteen terabytes of pooled HBM, the “behaves like one enormous GPU” rack from the NVIDIA section, rented whole.
Rough on-demand prices to anchor the tiers (us-east-1, approximate and subject to change; spot runs 30 to 60 percent cheaper):
| Instance | GPU | GPUs | VRAM each | ~ On-demand |
|---|---|---|---|---|
| g5.48xlarge | A10G | 8 | 24 GB | ~$16/hr |
| g6e.48xlarge | L40S | 8 | 48 GB | ~$30/hr |
| g7e.8xlarge | RTX PRO 6000 | 1 | 96 GB | ~$5/hr |
| g7e.48xlarge | RTX PRO 6000 | 8 | 96 GB | ~$33/hr |
| p4d.24xlarge | A100 | 8 | 40 GB | ~$33/hr |
| p5.48xlarge | H100 | 8 | 80 GB | ~$55-70/hr |
| p5e.48xlarge | H200 | 8 | 141 GB | ~$85/hr |
| p6-b200.48xlarge | B200 | 8 | 180 GB | ~$114/hr |
The thing that ties the p tier together, and that the g tier historically lacked, is EFA. NVLink solves talking inside one box; EFA, the Elastic Fabric Adapter, solves talking between boxes. It is a network card that can read and write GPU memory directly, without bouncing through the host, which is exactly what multi-node tensor parallelism and disaggregated prefill-and-decode need in order to move activations and KV cache between machines without falling over. Its bandwidth climbs with the tier, from 400 gigabits on the old A100 nodes to over three terabits on the Blackwell ones and nearly thirty on a full UltraServer. The number to know for 2026 is that g7e is the first cost-optimized instance ever to carry EFA, which means multi-node serving, once strictly a p-family privilege, is now available a tier down. The older g5, g6, and g6e have no EFA at all and simply cannot span nodes; on those, one instance is your ceiling.
Then there is the part only AWS has: its own chips. Inferentia and Trainium are AWS-designed accelerators, the current ones being Inferentia2 for inference and Trainium2 and Trainium3 for both training and serving, the newest of which multiplies in the same microscaling four-bit format Blackwell uses. The pitch is straightforward: for a model they support, they cost meaningfully less per token than the equivalent NVIDIA instance. The catch is equally straightforward, and is the reason most teams do not start here: you do not program them with CUDA. You go through the Neuron SDK, which compiles your model ahead of time, and not every model off the shelf just runs; the newest architectures lag, and you inherit a smaller ecosystem. So Trainium is a cost play for a stable, high-volume model where the up-front engineering pays for itself, not a default.
In the three-lever frame, AWS does not add a lever so much as sell you all of them at a chosen price point. The g-versus-p letter picks your tier across all three, EFA decides whether you can grow past one box, and Trainium is a standing offer to trade CUDA convenience for a lower bill. For most teams the honest path is a g instance while a single box is enough, and a p instance the moment it is not, with Trainium a deliberate optimization for later.
Google is the odd one again, but differently from Apple: it does not sell you a chip at all. Its accelerator, the TPU, exists only inside Google Cloud, rented by the hour and never shipped to your rack. It is also the most purpose-built thing in this part, a chip designed from the start to do the one operation a transformer is mostly made of.
To see what makes it different, remember what a GPU actually is: a general-purpose parallel processor with thousands of small cores, big register files, caches, and schedulers, which happens to be very good at matrix multiplication. A TPU throws most of that generality away. At its heart is a systolic array, called the MXU, a fixed grid of multiply-accumulate cells that the data flows through: you load a tile of weights into the grid once, then stream activations across it so each loaded weight is reused for a whole row of multiplications before it is touched again. That is the same “keep the running sum next to the math and stop going back to memory” idea from the tensor core in part 1, taken to its structural extreme. The payoff is very high efficiency on dense matmuls with little of a GPU’s control overhead; the cost is flexibility, which is why a TPU wants its work handed to it as one big compiled program rather than a pile of hand-written kernels.
Mapped to the three levers, the current inference-focused TPU, Ironwood, is strong across the board. On capacity it carries 192 GB of HBM per chip, right alongside a B200, and the more remarkable number is at pod scale: wire nine thousand of them together and the pod exposes something like 1.77 petabytes of memory as one coherent pool, a different order of thing from any single server. On compute the systolic array is genuinely efficient, and Ironwood is the first TPU to multiply FP8 natively; every generation before it did BF16 and INT8 only, and none of them, Ironwood included, does FP4, so on the precision ladder Google sits one rung behind Blackwell. On bandwidth Ironwood moves weights at about 7.4 TB/s, which puts it in the same class as a B200 for decode, while the cheaper, older TPUs are noticeably lighter and suit smaller or batch-heavy work.
Where Google genuinely leads is the thing NVIDIA’s rack only approximates: scale. TPUs are wired to their neighbors in a three-dimensional mesh, and the cubes of that mesh are stitched together not by ordinary switches but by optical ones that software can reconfigure on the fly, which lets Google carve out a slice of almost any size, route a dead chip out and a spare in without anyone noticing, and above all hold thousands of chips in a single coherent domain. NVIDIA’s biggest NVLink island tops out around seventy-two GPUs; a TPU pod coheres across thousands. For a model too large for any one node, or for serving a lot of long-context traffic at once, that is a real structural advantage.
The catches are the ones you would expect from custom silicon you can only rent. The native way to program it is not CUDA but XLA, a compiler that takes a whole model written in JAX and turns it into one optimized program, a different culture from the hand-tuned-kernel world the other vendors live in. The good news is you do not have to live there anymore: vLLM now runs on TPUs properly, not as the experiment it once was, so you can serve with the same API you would use on a GPU, and Google’s own Gemini is both trained and served on this hardware, about as strong a proof as exists that the stack works at the top end. The bad news is the ecosystem is still narrower than NVIDIA’s: fewer pre-tuned kernels, fewer third-party libraries, and day-one support for a brand-new open model almost always lands on CUDA first. And the whole thing lives on Google Cloud, so it is a bet on one provider. If that is a bet you have already made, and you are willing to work in JAX or a validated vLLM build, a TPU gives you strong capacity, bandwidth, and compute with scale-out nobody else matches. If you need CUDA and portability, Google will rent you NVIDIA GPUs in the same cloud, at the usual premium.
The software: inference engines
You have picked a chip. Now something has to actually run the model on it, and that something is an inference engine: the program that loads the weights, manages the KV cache, batches requests together, and turns your GPU into an HTTP endpoint. Everything from parts 1 and 2, the KV cache, prefill and decode, batching, quantization, is exactly what these engines exist to manage. Four of them cover almost every situation you will meet, and the useful way to tell them apart is not by benchmark charts, which change monthly, but by what each one is fundamentally for.
vLLM is the default, and you should treat it as the default. Its signature trick is PagedAttention: instead of handing each request one big contiguous slab of KV cache, it chops the cache into small fixed-size blocks and hands them out on demand, the way an operating system pages memory. That one idea removes almost all the reserved-but-unused cache that used to go to waste, which lets it keep far more requests in flight at once and pack the GPU to 80-odd percent utilization instead of the 30 or 40 you get from naive batching. On top of that it has the widest reach of any engine by a distance: NVIDIA, AMD, Intel, Google TPU, AWS Trainium, even plain CPUs, and it reads basically every model architecture and every quantization format worth having. You install it with pip, point it at a model, and get an OpenAI-compatible endpoint with no build step. It is not the single fastest thing you can run on an NVIDIA GPU, but it is the one that is fast enough, runs your model, and runs on your hardware, which is why it became the industry default. If you do not have a specific reason to pick something else, pick this.
llama.cpp is for running models locally. It is a different category from the other three: a single self-contained binary, no Python, no dependency tree, that will run a quantized model on almost anything, a CPU, a Raspberry Pi, and above all Apple Silicon through Metal. It authored the GGUF file format and the K-quant scheme you see everywhere (Q4_K_M and friends), and it can even split a model that does not fit in VRAM across GPU and CPU memory and run it anyway, slowly. If you have ever run a model on your own laptop you have almost certainly run llama.cpp, possibly without knowing it, because Ollama and LM Studio are friendly wrappers around its engine. What it is not built for is serving many users at once: it manages the KV cache in rigid per-request slots with none of vLLM’s paging, so throughput falls apart the moment you have more than a handful of concurrent requests. Local and single-user it wins on sheer simplicity; as a multi-tenant server it is the wrong tool.
TensorRT-LLM is for squeezing the last drop out of NVIDIA hardware. Where vLLM interprets your model on the fly, TensorRT-LLM compiles it ahead of time into a graph of CUDA kernels hand-tuned for one specific GPU, one batch-size range, and one sequence length, with the Python overhead stripped out entirely. Tuned well, on the same NVIDIA GPU it is meaningfully faster than vLLM, often something like half again the throughput at the same latency, which at scale is a real chunk of your bill. The price is rigidity: it is NVIDIA-only, and the classic workflow makes you recompile the engine, a slow step, every time the weights or the batch ceiling or the sequence length change, and its support for brand-new architectures lags. Recent work to drive it from a PyTorch front end is softening those edges, but the trade still holds: reach for it when the model is stable, you are running at real scale, and you have decided the tuning effort is worth shaving the bill. Not before.
SGLang is for workloads where requests share a lot of text. Its signature is RadixAttention: it keeps every request’s KV cache in a shared tree keyed by the actual tokens, so any two requests that begin the same way, the same long system prompt, the same document in a RAG pipeline, the same history in a multi-turn agent, automatically reuse each other’s already-computed cache instead of recomputing it. This is the prefix-caching idea from part 2 taken to its logical end, and on the workloads that suit it the effect is dramatic: cache-hit rates jump and time-to-first-token drops by multiples. It also enforces structured output, forcing the model to emit valid JSON against a schema, at almost no throughput cost, where other engines stall. On plain unique-prompt traffic it is roughly neck and neck with vLLM; on prefix-heavy, agentic, or structured traffic it pulls clearly ahead. If you are serving agents, RAG, or repeated schemas, this is the one to try.
The honest summary: the field has two general-purpose open servers, vLLM and SGLang, within a few percent of each other on raw throughput and differing mainly by workload shape; TensorRT-LLM sitting above them on NVIDIA alone when you pay the tuning tax; and llama.cpp off to the side, owning local. And the gaps close every month. Prefix caching, chunked prefill, continuous batching, FP8 and FP4 quantization, and even splitting prefill and decode across separate pools of GPUs are all showing up in every engine now. Pick by what you are actually serving, not by whichever benchmark screenshot is going around this week.
How to pick
Put the three levers back in order and the whole landscape collapses into a short series of questions. First, capacity: does the model, its KV cache, and a little room actually fit on the memory you can afford? This is a hard gate; get it wrong and nothing else matters. If it does not fit on one chip, either quantize it down until it does or accept that you are paying the interconnect tax to spread it. Second, bandwidth: how fast do you need to decode, because that number is set almost entirely by memory bandwidth, and it is where the HBM-versus-GDDR-versus-unified-memory choice actually bites. Third, compute and precision: check that the chip natively multiplies the format you plan to quantize to, or your quantization saves memory and buys no speed. Fourth, and only if a model outgrows a single chip, interconnect: NVLink and NVSwitch if you are spreading one model wide, EFA-class networking if you are spreading it across machines.
As for who to buy from, the honest defaults. NVIDIA if you want the path of least resistance and the deepest software, which is most people. AMD if you want the same serving for less money and can live with a slightly rougher edge and a weaker multi-GPU story until its rack ships. Apple, DGX Spark, or a Strix Halo box if the job is running a big model on your own hardware rather than serving a crowd. Google’s TPUs if you already live on Google Cloud and will work in its software. And AWS as the place you rent most of the above, starting on a g instance and moving to a p instance the day one box stops being enough. On top of any of them, vLLM until you have a specific reason to reach for something sharper.
None of this is really about which chip is biggest. It is about reading a chip as three numbers and a wire, matching them to a model and a workload, and paying for exactly that and no more. Which, now that the model is demystified and the metal is demystified, is the whole job.
Comments & reactions
React or leave a comment below — sign in with GitHub. It all lives in this site's GitHub Discussions.