gpt-compute

LLM VRAM Calculator

Qwen3.8 27BNew

Fits on RTX 4090 24GB

Memory
18.7 / 24GiB
Speed
40tok/s
Download
17.6GB

Run it with llama.cpp

  1. 1

    Install

  2. 2

    Paste into a terminal

    llama.cpp command
    llama-server \  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL \  --ctx-size 32768 \  --n-gpu-layers 999 \  --flash-attn on \  --temp 1 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0 --repeat-penalty 1 \  --no-mmproj \  --jinja \  --host 0.0.0.0 --port 8080
    Show 1 more note
    • Sampling: Unsloth's recommended “Thinking” settings for Qwen3.8 27B.
  3. 3

    Chat

    When the log says it is listening on port 8080, open http://localhost:8080 in your browser.

Runs on 36 of 39 machines

Consumer GPUs
MacBooks
Workstation
Unified memory
Datacenter

Too small: 3 more.

Cheapest to rent: at about $0.11/h, 16 tok/s. Prices of 2026-11-01. Data: FastGPU GPU cloud pricing dataset, CC BY 4.0; setup prices derived by gpt-compute.

How the numbers are calculated

Most calculators multiply parameters by a nominal bit width and guess the rest. These are the formulas used here instead — the code is open and every constant is documented next to its use.

Weights: real file sizes, not bits × params

A Q4_K_M file holds 5 bits per weight, not 4Qwen3.8 27B: 26.9 billion parameters at 4 bits would be 13.4 GB. The real Q4_K_M file is 16.8 GB: the 4-bit values plus 0.5 bit per weight of block scales and 0.5 bit of tensors kept at higher precision. With llama.cpp on a discrete GPU the 0.8 GB token-embedding table stays in system RAM, so 16.0 GB of weights go to VRAM.26.9B params × 4 bits = 13.4 GBQwen3.8 27B Q4_K_M · 5.0 bits per weight16.8 GB4 bits per weight+0.5 bit block scales+0.5 bit tensors kept at higher precisionIn VRAM (llama.cpp, discrete GPU)16.0 GBtoken embeddings → system RAM, 0.8 GB

Q4_K_M is not 4 bits per weight. Block scales and the tensors kept at higher precision push it to roughly 4.8–5.1 bits, and “dynamic” quants vary per model. The calculator uses the measured size of every GGUF file, FP8 / NVFP4 / INT4 checkpoint and MLX conversion published on Hugging Face (only vLLM's on-the-fly FP8 is estimated from the BF16 checkpoint).

llama.cpp keeps the token-embedding table in system RAM on discrete GPUs, so it is excluded from VRAM there.

KV cache: per layer type, from config.json

What one layer caches as the context grows, by layer typeEach box is one token of context. Full attention (GQA) caches every token, 2,048 values per token for 8 KV heads of 128 (Llama 3.1 8B). MLA also caches every token but only a 576-value latent (DeepSeek V3.1, Kimi K2.6). DeepSeek V4-style compressed attention stores one 512-value entry per 4 tokens. A sliding-window layer keeps only the most recent tokens. Linear-attention and Mamba layers keep one fixed-size state, the same at any context length.Full attention (GQA)MLA latentCompressed 4×Sliding windowLinear attn / Mambaone fixed-size state, any lengthcontext growsone box per token

GQA: 2 × kv_heads × head_dim × bytes × tokens

MLA: (kv_lora_rank + rope_dim) × bytes × tokens

Sliding-window layers cache only the window (plus one 512-token batch in llama.cpp). Linear-attention and Mamba layers keep a fixed-size state instead of a cache. DeepSeek V4-style compressed attention stores one entry per 2, 4 or 128 tokens. head_dim is read from the config; hidden ÷ heads is only the fallback when a config doesn't give it.

Runtime overhead

llama.cpp allocates what you ask for; vLLM takes 90% and fills the rest with KV cacheLlama 3.1 8B with 32k context on one 24 GiB RTX 4090. llama.cpp (UD-Q4_K_XL) uses 0.76 GiB of runtime memory (0.45 GiB CUDA context and a 0.31 GiB compute buffer), 4.35 GiB of weights and 4 GiB of KV cache for one 32k sequence, leaving 14.9 GiB free. vLLM (FP8) reserves 90%, 21.6 GiB: 1.8 GiB for the CUDA context, CUDA graphs and an 8,192-token profiling pass, 8.46 GiB of weights, and the remaining 11.4 GiB becomes KV cache, room for 2 sequences of 32k tokens.Llama 3.1 8B · 32k context · one RTX 4090, 24 GiBllama.cpp UD-Q4_K_XLweightsKV 32k14.9 GiB freeCUDA context 0.45 + compute buffer 0.31 GiBvLLM FP890% = 21.6 GiBweights32k32kCUDA context + graphs + 8,192-token pass 1.8 GiBKV: 2 × 32k sequences fit

llama.cpp: ~0.45 GiB CUDA context per GPU (less on Macs) plus a compute buffer of 512 tokens × (vocab + 8 × hidden) × 4 bytes. Context is shared by all --parallel slots.

vLLM: reserves 90% of each GPU, profiles an 8,192-token forward pass and captures CUDA graphs; the remainder becomes KV cache. The calculator reports how many sequences of your context length fit.

Decode speed: memory-bandwidth bound

Time to generate one token on an RTX 4090, for a dense model, an MoE model and an MoE model with offloaded expertsThe RTX 4090 has 1,008 GB/s of memory bandwidth; at an efficiency of 0.72 that is 726 GB/s. Bars show the time per token with 32k tokens of context, to scale. Qwen3.8 27B is dense: reading 16.7 GB of weights takes 23 ms, plus 3.2 ms to read the KV cache and 1.6 ms of per-layer latency, 28 ms in all or 36 tokens per second. Qwen3.6 35B-A3B reads only 8 of its 256 experts per token, 2.1 GB, so a token takes 5.7 ms, 177 tokens per second; reading all 22 GB of its weights would take 30 ms. gpt-oss 120B keeps the experts of 24 of its 36 layers in system RAM: the 1.25 GB it reads from there per token, at half of 80 GB/s, takes 31 ms, so a token takes 41 ms, 24 tokens per second.RTX 4090 · 1,008 GB/s × η 0.72 = 726 GB/sQwen3.8 27B · dense28 ms → 36 tok/s16.7 GB ÷ 726 GB/s = 23 msKV cache read at 32k contextper-layer latencyQwen3.6 35B-A3B · reads 8 of 256 experts5.7 ms → 177 tok/sreading all 22 GB would take 30 msgpt-oss 120B · experts of 24/36 layers in RAM41 ms → 24 tok/s1.25 GB ÷ (80 GB/s RAM × 0.5) = 31 ms

time per token ≈ (active weight bytes + KV bytes read) ÷ (η × bandwidth) + per-layer latency

MoE models read only the active experts; with B concurrent sequences the share of experts touched per step is 1 − (1 − k/E)B. llama.cpp splits layers across GPUs, which adds memory but not single-stream bandwidth; vLLM tensor parallelism adds both, minus all-reduce latency. Offloaded experts are read at about half the system-RAM bandwidth.

Prefill and time to first token

Time to first token grows faster than the prompt, and far faster on a MacLlama 3.1 8B with llama.cpp, prompts from 0 to 128k tokens. Reading a 128k-token prompt takes 44 seconds on an RTX 4090 and about 12 minutes on an M4 Max 128GB, which has 17 times less usable compute. The curves bend upward: under the M4 Max curve, the part proportional to the prompt (2 × active params × prompt) is overtaken by attention, which grows with the square of the prompt and is about 70% of the time at 128k.prompt length64k128kM4 Max 128GB12 minRTX 409044 sLlama 3.1 8B · llama.cpp2 × active params × promptattention (∝ prompt²)

FLOPs ≈ 2 × active params × prompt + attention (∝ prompt²)

Prompt processing is compute bound (the prompt² term applies to full-attention layers; sliding-window and linear layers grow linearly). Apple Silicon generates quickly but prefills slowly, so a long prompt can take minutes before the first token — the calculator shows it. M5 Macs prefill faster: their GPU Neural Accelerators add about 2.5× in llama.cpp and 3.3× with MLX weights in any app (LM Studio's, Ollama's and Unsloth's llama.cpp builds aren't counted until they use them).

Where the models come from

How models get into the calculatorCandidates come from Unsloth on Hugging Face and Unsloth's catalog, Ollama's library and LM Studio's catalog. Each must map to one upstream release, have a config.json the calculator can parse, and file sizes that fit its parameter count. Models that fail are held back; the rest reach models.json through a weekly pull request, and the calculator reads that file.UnslothHugging Face + catalogOllamalibraryLM Studiocatalogmaps to one upstream releaseconfig.json parsessizes fit the parameter countheld backweekly pull requestmodels.jsoncalculator

Next to a hand-curated core, models are discovered from Unsloth on Hugging Face and Unsloth's model catalog, Ollama's library and LM Studio's catalog, then checked: each must map to a single upstream release, have a config the calculator can parse, and file sizes consistent with its parameter count. Recommended sampling settings come from Unsloth's run guides, entered by hand and linked to each guide.

Ollama tags are sized from Ollama's registry manifests (weights, vision projector, drafter); LM Studio entries from the Hugging Face repos their model.yaml points to.

Units and limits

File sizes in GB, memory in GiB, speeds within ±15%Two rulers measure the same bytes. The Qwen3.8 27B UD-Q4_K_XL download is 17.6 GB, counted in GB of 10^9 bytes, and the same file is 16.4 GiB, counted in GiB of 2^30 bytes. An RTX 4090 sold as 24GB has 24 GiB of memory, which is 25.8 GB. On that card the calculator estimates about 40 tokens per second for this file; expect 34 to 46.1 GB = 10⁹ bytes · file sizes17.6 GB25.8 GBQwen3.8 27B · UD-Q4_K_XLRTX 4090 “24GB”1 GiB = 2³⁰ bytes · memory16.4 GiB24 GiBSpeed on the RTX 4090 ≈ 40 tok/s±15%: 34–46 tok/s

Memory is shown in GiB (as nvidia-smi does); file sizes in GB. Speeds are estimates calibrated to public llama.cpp benchmarks — expect ±15%, more for multi-GPU setups and brand-new architectures, which are flagged in the calculator’s Model card tab.

Frequently asked questions

Why does my “24 GB” GPU show less usable memory?+

GPU memory is sold in binary gigabytes: a 24 GB RTX 4090 reports 24,564 MiB, i.e. 24 GiB. Model files on Hugging Face are listed in decimal GB (10⁹ bytes), which is about 7% smaller per unit. This calculator works in bytes and shows memory in GiB — the same unit nvidia-smi uses — and subtracts the CUDA/Metal context each GPU needs before anything loads.

Should I quantize the KV cache?+

Q8_0 KV roughly halves cache memory with no measurable quality loss for most models and is the easiest way to fit a longer context. Q4_0 quarters it but can hurt long-context recall on some models. In llama.cpp a quantized V cache requires flash attention, which the generated commands enable. vLLM supports FP8 KV.

Why does vLLM use almost all of my VRAM?+

vLLM pre-allocates gpu_memory_utilization (90% by default) of every GPU, loads the weights, profiles activation memory, and turns the rest into paged KV-cache blocks. The useful question for vLLM is therefore how many tokens of KV fit — which the calculator reports as concurrent sequences at your context length.

What is MoE expert offload?+

Mixture-of-experts models only use a few experts per token. llama.cpp can keep the attention and shared weights on the GPU and put the routed experts of some layers in system RAM (--n-cpu-moe). Because only the active experts are read per token, a 120B MoE can run at usable speeds on a single 24 GB card with enough RAM — the calculator picks the smallest --n-cpu-moe that fits.

Where does the model list come from?+

A weekly job reads three catalogs: Unsloth's Hugging Face organization and model catalog, Ollama's model library and LM Studio's model catalog. New official releases (at least 3B parameters, not fine-tunes of someone else's model) are added automatically once their config passes sanity checks; Unsloth's run guides supply the recommended sampling settings. Every number is still computed from the model's own config.json and the real file sizes.

What happens when I paste a Hugging Face link?+

A link to one of the listed models or its downloads opens it with that file selected. A quantized copy or fine-tune of a listed model (a bartowski GGUF, an mlx-community conversion, …) is found in a list the weekly job builds from Hugging Face: the calculator keeps the model's architecture and uses the copy's real file sizes. Any other repo can be looked up live: choose “Look up … on Hugging Face” and your browser asks the Hugging Face API for that repo's metadata, file list and config.json. Only the repo name is sent, without cookies, and only when you choose to look it up or open a link to such a repo. A model new to the calculator is marked approximate: its architecture comes straight from its config.json, without the checks listed models get, and speeds aren't calibrated for it.

Why are Ollama's and LM Studio's downloads a different size?+

They are different files. Ollama's library tags are its own GGUF builds, sized from Ollama's registry: some keep a few tensors at higher precision, bundle the vision encoder or a speculative-decoding drafter, and Ollama always loads a tag's vision projector. LM Studio's catalog downloads lmstudio-community's GGUF and MLX conversions. The calculator lists each source with its measured size, so pick the file you will actually run.

What are Unsloth Dynamic (UD) quants?+

Unsloth's GGUFs keep the layers that are most sensitive to quantization at higher precision and squeeze the rest, so a UD-Q4_K_XL is usually a little larger than a standard Q4_K_M but closer to the full-precision model. Unsloth recommends UD-Q4_K_XL as the default and at least UD-Q2_K_XL for agentic use. Both kinds are listed with their real file sizes.

Where do the rental prices come from?+

From FastGPU's GPU cloud pricing dataset (https://fastgpu.co/dataset, CC BY 4.0), which records daily on-demand and spot prices across cloud providers. The weekly data refresh takes the latest day: a setup's price is the median on-demand price per GPU across providers times the number of GPUs, and the tooltip adds the cheapest single offer. Prices are per hour, before storage and egress, and only for GPUs one can rent; Macs and desktop Radeon cards have none.

How accurate are the speed numbers?+

Decode speed is modeled as memory-bandwidth bound with per-device efficiency calibrated against llama.cpp's public benchmarks, plus per-layer kernel latency. Expect ±15% for common setups; multi-GPU and brand-new architectures are less certain. Treat them as planning numbers and check your own logs.