Same Weights, Different Stack
We served three open mixture-of-experts models on one AMD MI350X and one NVIDIA B300, with two serving engines, and measured them the same way each time. The chip matters, but on these models the engine and its version decided as much: how fast a model served, and whether it served at all.
00The setup
Two workloads: generation (about 110 tokens in, 1,000 out, so decode dominates) and summarization (about 1,010 in, 100 out, so prefill matters). The third run repeated every point three times, which lets us say how big a difference has to be before it is real.
01First, the noise floor
Across three repeats of every point in the third run, the spread (max minus min, over the median) was 0.3 % at the median and 5.6 % at worst. The same model on the same engine on two different MI350X machines two days apart agreed within 1 %. So in what follows, differences under about 6 % are noise; the ones that matter are several times that.
02Whether a model runs depends on the stack
Before speed, the plainer question. Each cell is what we observed on that engine version and build; none of it says a model is broken, only that this combination did not serve it.
| model | vLLM 0.27 ROCm | SGLang 0.5.20 ROCm | vLLM 0.27 CUDA | vLLM 0.30 CUDA | SGLang 0.5.20 CUDA |
|---|---|---|---|---|---|
| Qwen3-30B-A3B | serves | serves | serves | serves | not run |
| Nemotron-3-Nano-30B-A3B | serves | did not startdefault MoE backend imports an NVIDIA-only library; with the Triton backend, a Mamba convolution kernel is missing in this build | serves | serves | serves |
| Instella-MoE-16B-A3B | did not startreshape error while compiling the attention | did not startCUDA-graph capture failed in the KV-cache kernel | did not startsame reshape error | refused"does not dispatch through the Transformers attention interface" | did not startsame CUDA-graph capture error |
- Instella failed the same way on AMD and on NVIDIA for each engine. Its attention module computes attention itself instead of calling the standard pluggable interface, so a generic serving backend has nowhere to substitute its paged-attention kernel. That is a software-support gap, not a silicon one, and it held on AMD's own hardware.
- A one-line patch routing that call through the standard interface did not get past vLLM 0.27's compile step on either vendor. AMD publishes its own serving recipe for Instella (an SGLang overlay); our launcher failed to start it, so that path is untested here, not failed.
- Nemotron served everywhere except SGLang's ROCm build, where its Mamba layers needed a kernel that build did not have. On SGLang's CUDA build it served fine.
If a model card says "supports vLLM and SGLang", read it as "on some versions, on some backends". Test the exact engine, version and GPU vendor you will deploy.
03Same engine on both cards
An earlier run compared vLLM 0.27 on the MI350X with vLLM 0.30 on the B300, which mixed hardware with software. The third run put vLLM 0.27.1 on both. Generation throughput, all requests together:
| MI350X | B300 | B300 ÷ MI350X | earlier, mixed versions | |
|---|---|---|---|---|
| Nemotron, concurrency 1 | 227 tok/s | 427 tok/s | 1.88× | 1.82× |
| Nemotron, concurrency 32 | 4,218 | 8,031 | 1.90× | 1.82× |
| Qwen, concurrency 1 | 208 | 320 | 1.54× | 1.54× |
| Qwen, concurrency 32 | 4,433 | 6,652 | 1.50× | 1.52× |
NVIDIA's own model gains about 25 % more from moving to NVIDIA than the neutral model does, and that held once the engine version was the same. The likely reason is kernel maturity: Nemotron's Mamba layers have long-tuned CUDA kernels and newer ROCm ports. It is still a stack comparison, not a silicon one: same engine version, but not the same kernels underneath.
04Swapping the engine moves throughput by up to a quarter, in either direction
SGLang 0.5.20 throughput relative to vLLM 0.27.1 on the same card and model, median of three repeats; both engines at their defaults.
| card, model, workload | concurrency | vLLM | SGLang | time to first token, vLLM → SGLang |
|---|---|---|---|---|
| MI350X, Qwen, generation | 32 | 4,433 tok/s | 5,451 tok/s | 71 → 98 ms |
| MI350X, Qwen, generation | 1 | 208 | 230 | 17 → 40 ms |
| B300, Nemotron, summarization | 32 | 4,500 | 3,842 | 202 → 310 ms |
The pattern: SGLang started every request later (higher time to first token, in all 12 paired cells) and, in all but one, produced tokens faster once a request was running (lower inter-token latency). Long outputs reward the second; long prompts at high concurrency punish the first. The exception is the case where SGLang lost: Nemotron summarization at concurrency 32 on the B300, slower on both counts. So the same engine swap gained 24 % on one model and workload and lost 15 % on another.
For scale: the engine swap moved throughput by up to 24 %, upgrading vLLM from 0.27 to 0.30 on the B300 moved it by at most 5 %, and run-to-run noise was 0.3 %.
05What we did not measure
- Whether the engines produce the same tokens. Our determinism probe in these runs used a target prompt that any model answers by repeating it, so it could not have seen divergence; we withdrew those results and fixed the probe, which has not been re-run. The claim that the same weights on a different engine can give different outputs, made in post 50, rests on the literature, not on these runs.
- TensorRT-LLM. The third run was stopped before it reached it, and before Qwen on SGLang's CUDA build.
- Tuning. Every engine ran close to its defaults. A tuned deployment would move these numbers; the point here is how far the defaults alone differ.
- Scale. One GPU per vendor, concurrency up to 32, one day per run.
06What to take from it
- Pick a stack, not a chip. On these models the hardware gap was 1.5-1.9×, the engine gap up to ±24 %, and a model's support could be all-or-nothing on a given engine build.
- Test the exact combination. Model × engine × version × GPU vendor. A "supported" label doesn't cover all four.
- Know your workload's shape before choosing an engine. Long outputs and long prompts favoured different engines here.
- Repeat before you compare. Three repeats cost a few minutes and turned "faster" into "faster by more than the noise".
Measured with llm-masters projects/16 (GenAI-Perf definitions), three runs on 2026-09-29 and 2026-09-30. Raw summaries, versions and container digests are in the llm-masters repository under benchmarks/results/aigenperf.