Data Lab LLM API
Single-request decode speed (tokens per second) of every model on this CPU cluster versus common GPUs. The cluster column is measured; GPU columns are bandwidth-roofline estimates.
| model | weights GB bf16 / int8 / int4 | this cluster measured tok/s | Quadro RTX 6000 (lab, g-01) 24 GB, 672 GB/s | RTX 3090 24 GB, 936 GB/s | RTX 4090 24 GB, 1,008 GB/s | RTX 5090 32 GB, 1,792 GB/s | L40S 48 GB, 864 GB/s | RTX 6000 Ada 48 GB, 960 GB/s | RTX PRO 6000 Blackwell 96 GB, 1,792 GB/s | A100 40GB 40 GB, 1,555 GB/s | A100 80GB 80 GB, 2,039 GB/s | H100 SXM 80 GB, 3,350 GB/s | H200 141 GB, 4,800 GB/s | B200 180 GB, 8,000 GB/s |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
qwen3-8b8B | 16.4 / 8.7 / 4.6 | 6.4 live, 24 h, 8 req | 32.8 bf16 | 45.7 bf16 | 49.2 bf16 | 87.4 bf16 | 42.1 bf16 | 46.8 bf16 | 87.4 bf16 | 75.9 bf16 | 99.5 bf16 | 163 bf16 | 234 bf16 | 390 bf16 |
qwen3.8-27b27B | 54.0 / 28.6 / 15.1 | 3.6 benchmark, 4 node(s) | 35.6 int4 | 49.5 int4 | 53.3 int4 | 94.8 int4 | 24.2 int8 | 26.8 int8 | 26.5 bf16 | 43.5 int8 | 30.2 bf16 | 49.6 bf16 | 71.1 bf16 | 118 bf16 |
qwen3-30b-a3b30B, 3.3B active | 61.0 / 32.3 / 17.1 | 3.4 live, 24 h, 69 req | 291 int4 | 405 int4 | 436 int4 | 776 int4 | 198 int8 | 220 int8 | 217 bf16 | 356 int8 | 247 bf16 | 406 bf16 | 582 bf16 | 970 bf16 |
qwen3-32b33B | 65.6 / 34.8 / 18.4 | 10.7 benchmark, 6 node(s) | 29.3 int4 | 40.8 int4 | 43.9 int4 | 78.0 int4 | 19.9 int8 | 22.1 int8 | 21.9 bf16 | 67.7 int4 | 24.9 bf16 | 40.9 bf16 | 58.5 bf16 | 97.6 bf16 |
qwen3.6-35b-a3b35B, 3B active | 70.0 / 37.1 / 19.6 | 21.2 benchmark, 1 node(s) | 320 int4 | 446 int4 | 480 int4 | 853 int4 | 217 int8 | 242 int8 | 239 bf16 | 740 int4 | 272 bf16 | 447 bf16 | 640 bf16 | 1,067 bf16 |
qwen3-235b-a22b235B, 22B active | 470 / 249 / 132 | 11.0 benchmark, 4 node(s) | 1.1 1 card, 15% in VRAM 227 on 8 cards | 2.4 1 card, 15% in VRAM 316 on 8 cards | 2.4 1 card, 15% in VRAM 340 on 8 cards | 5.1 1 card, 20% in VRAM 605 on 8 cards | 2.9 1 card, 31% in VRAM 168 on 4 cards | 2.9 1 card, 31% in VRAM 187 on 4 cards | 10.7 1 card, 64% in VRAM 198 on 2 cards | 2.7 1 card, 26% in VRAM 303 on 4 cards | 4.3 1 card, 53% in VRAM 225 on 2 cards | 8.5 1 card, 53% in VRAM 370 on 2 cards | 64.1 1 card, 95% in VRAM 530 on 2 cards | 520 int4 |
qwen3-coder-480b480B, 35B active | 960 / 509 / 269 | 5.3 benchmark, 4 node(s) | 0.70 1 card, 7% in VRAM 121 on 16 cards | 1.4 1 card, 7% in VRAM 169 on 16 cards | 1.4 1 card, 7% in VRAM 182 on 16 cards | 2.8 1 card, 10% in VRAM 323 on 16 cards | 1.5 1 card, 15% in VRAM 183 on 8 cards | 1.5 1 card, 15% in VRAM 204 on 8 cards | 3.7 1 card, 31% in VRAM 219 on 4 cards | 1.5 1 card, 13% in VRAM 330 on 8 cards | 1.7 1 card, 26% in VRAM 250 on 4 cards | 3.4 1 card, 26% in VRAM 410 on 4 cards | 4.7 1 card, 46% in VRAM 588 on 4 cards | 6.2 1 card, 60% in VRAM 555 on 2 cards |
deepseek-v3.1671B, 37B active | 1,342 / 711 / 376 | 4.6 benchmark, 4 node(s) | 0.60 1 card, 5% in VRAM 115 on 32 cards | 1.3 1 card, 5% in VRAM 160 on 32 cards | 1.3 1 card, 5% in VRAM 172 on 32 cards | 2.6 1 card, 7% in VRAM 306 on 16 cards | 1.3 1 card, 11% in VRAM 147 on 16 cards | 1.3 1 card, 11% in VRAM 164 on 16 cards | 3.1 1 card, 22% in VRAM 360 on 8 cards | 1.3 1 card, 9% in VRAM 265 on 16 cards | 1.5 1 card, 19% in VRAM 409 on 8 cards | 3.0 1 card, 19% in VRAM 673 on 8 cards | 3.6 1 card, 33% in VRAM 556 on 4 cards | 4.2 1 card, 43% in VRAM 927 on 4 cards |
kimi-k2.61,030B, 32B active | 2,060 / 1,092 / 577 | 2.7 benchmark | 0.70 1 card, 3% in VRAM 133 on 32 cards | 1.4 1 card, 3% in VRAM 185 on 32 cards | 1.4 1 card, 3% in VRAM 199 on 32 cards | 2.9 1 card, 5% in VRAM 354 on 32 cards | 1.5 1 card, 7% in VRAM 170 on 16 cards | 1.5 1 card, 7% in VRAM 189 on 16 cards | 3.2 1 card, 15% in VRAM 416 on 8 cards | 1.5 1 card, 6% in VRAM 307 on 32 cards | 1.6 1 card, 12% in VRAM 402 on 16 cards | 3.2 1 card, 12% in VRAM 661 on 16 cards | 3.5 1 card, 22% in VRAM 1,114 on 8 cards | 3.8 1 card, 28% in VRAM 1,071 on 4 cards |
qwen3.8-max (2.4T)2,450B, 95B active | 4,900 / 2,597 / 1,372 | 0.66 benchmark | 0.20 1 card, 1% in VRAM 44.7 on 128 cards | 0.50 1 card, 1% in VRAM 62.2 on 128 cards | 0.50 1 card, 1% in VRAM 67.0 on 128 cards | 1.0 1 card, 2% in VRAM 119 on 64 cards | 0.50 1 card, 3% in VRAM 57.4 on 64 cards | 0.50 1 card, 3% in VRAM 63.8 on 64 cards | 1.0 1 card, 6% in VRAM 119 on 32 cards | 0.50 1 card, 2% in VRAM 103 on 64 cards | 0.50 1 card, 5% in VRAM 136 on 32 cards | 1.0 1 card, 5% in VRAM 223 on 32 cards | 1.0 1 card, 9% in VRAM 319 on 16 cards | 1.1 1 card, 12% in VRAM 532 on 16 cards |
GPU numbers are a roofline estimate, the same method as the dashboard's A100 references:
tok/s = eff * memory bandwidth / bytes read per token
using the best precision whose weights fit in 90% of VRAM (bf16 > int8 > int4). Batch-1 decode
on a GPU is memory-bandwidth bound, so this is an upper bound; vLLM / llama.cpp usually land at
50-80% of it for dense models and 30-60% for mixture-of-experts (MoE) models, whose many small
expert matrices use the GPU less efficiently. Two views per GPU:
one card: if the model does not fit, the resident share of the int4 weights is read from VRAM
and the rest streamed over PCIe every token (the dashboard's A100 references use this);
multi-GPU: the smallest power-of-two group that holds it at int4. Up to 8 GPUs (one server)
bandwidth adds up with tensor-parallel losses; beyond that the extra servers run as pipeline
stages, which adds capacity but not single-stream speed.
CPU numbers are measured (benchmarks in results/*.jsonl, live API requests in the gateway DB).
The cluster is 16 CPU servers (Xeon 8260, 504 GB RAM each, no GPUs). Its advantage is capacity: it holds models of hundreds of billions to trillions of parameters that need 8-32 data-center GPUs, and serves many requests at once. Single-request speed on small models is below a GPU's; that is memory-bandwidth physics. Live engineering progress: cpullm.projects.data.syr.edu (login required).