Lab · LLM VRAM · SmolLM
ComputedSmolLM2 135M Instruct memory requirements
135M parameters — 257 MiB of weights at FP16, 101 MiB at Q4_K_M, and 22.5 KiB of KV cache per token on top. Solved against 22 real accelerators; 22 of them can hold it.
Parameters
135M
134,515,008 exactly
Weights, Q4_K_M
101 MiB
257 MiB unquantized
KV cache
22.5 KiB
per token · 30L × 3 KV × 64
Smallest card (Q4)
8 GB
Jetson Orin Nano Super (8GB)
Weights at each precision
| Precision | Bits/weight | Weights | Source | Notes |
|---|---|---|---|---|
| FP16 / BF16 | 16.00 | 257 MiB | measured file | Unquantized. Two bytes per parameter, exactly. |
| Q8_0 | 8.61 | 138 MiB | measured file | Effectively lossless; the usual reference point for quantized quality. |
| Q6_K | 8.23 | 132 MiB | measured file | Quality loss is hard to measure on most benchmarks. |
| Q5_K_M | 6.67 | 107 MiB | measured file | A middle point when Q4 fits with too little room for context. |
| Q4_K_M | 6.27 | 101 MiB | measured file | The default choice for local inference — the best size/quality knee. |
| Q3_K_M | 5.56 | 89 MiB | measured file | Measurable quality loss. Worth it only to make a model fit at all. |
Every accelerator
Largest quant that fits with at least 4k of context, the context it leaves, and the bandwidth ceiling on decode speed. Each row links to the worked page for that pairing.
| Accelerator | Memory | Bandwidth | Best quant | Max context | Ceiling |
|---|---|---|---|---|---|
| Apple M3 Ultra (512GB) | 512 GB | 819 GB/s | FP16 / BF16 | 8,192 | 1789 tok/s |
| H200 141GB (SXM) | 141 GB | 4800 GB/s | FP16 / BF16 | 8,192 | 10485 tok/s |
| Apple M4 Max (128GB) | 128 GB | 546 GB/s | FP16 / BF16 | 8,192 | 1193 tok/s |
| H100 80GB (SXM5) | 80 GB | 3350 GB/s | FP16 / BF16 | 8,192 | 7318 tok/s |
| A100 80GB (SXM) | 80 GB | 2039 GB/s | FP16 / BF16 | 8,192 | 4454 tok/s |
| Jetson AGX Orin (64GB) | 64 GB | 204.8 GB/s | FP16 / BF16 | 8,192 | 447 tok/s |
| RTX 6000 Ada Generation | 48 GB | 960 GB/s | FP16 / BF16 | 8,192 | 2097 tok/s |
| L40S | 48 GB | 864 GB/s | FP16 / BF16 | 8,192 | 1887 tok/s |
| Apple M4 Pro (48GB) | 48 GB | 273 GB/s | FP16 / BF16 | 8,192 | 596 tok/s |
| A100 40GB (SXM) | 40 GB | 1555 GB/s | FP16 / BF16 | 8,192 | 3397 tok/s |
| GeForce RTX 5090 | 32 GB | 1792 GB/s | FP16 / BF16 | 8,192 | 3914 tok/s |
| GeForce RTX 4090 | 24 GB | 1008 GB/s | FP16 / BF16 | 8,192 | 2202 tok/s |
| Radeon RX 7900 XTX | 24 GB | 960 GB/s | FP16 / BF16 | 8,192 | 2097 tok/s |
| GeForce RTX 3090 | 24 GB | 936 GB/s | FP16 / BF16 | 8,192 | 2045 tok/s |
| Apple M4 (24GB) | 24 GB | 120 GB/s | FP16 / BF16 | 8,192 | 262 tok/s |
| GeForce RTX 5080 | 16 GB | 960 GB/s | FP16 / BF16 | 8,192 | 2097 tok/s |
| GeForce RTX 5070 Ti | 16 GB | 896 GB/s | FP16 / BF16 | 8,192 | 1957 tok/s |
| GeForce RTX 4080 SUPER | 16 GB | 736 GB/s | FP16 / BF16 | 8,192 | 1608 tok/s |
| GeForce RTX 4070 Ti SUPER | 16 GB | 672 GB/s | FP16 / BF16 | 8,192 | 1468 tok/s |
| GeForce RTX 4060 Ti 16GB | 16 GB | 288 GB/s | FP16 / BF16 | 8,192 | 629 tok/s |
| GeForce RTX 3060 12GB | 12 GB | 360 GB/s | FP16 / BF16 | 8,192 | 786 tok/s |
| Jetson Orin Nano Super (8GB) | 8 GB | 102 GB/s | FP16 / BF16 | 8,192 | 223 tok/s |
Questions about this model
How much VRAM does SmolLM2 135M Instruct need?
257 MiB for the weights at FP16 and 101 MiB at Q4_K_M, from an exact count of 134,515,008 parameters. On top of that, the KV cache costs 22.5 KiB per token of context — 180 MiB at 8k and 720 MiB at 32k.
What is the smallest GPU that runs SmolLM2 135M Instruct?
Of the 22 accelerators here, the smallest that holds Q4_K_M weights is the Jetson Orin Nano Super (8GB) at 8 GB — 101 MiB of weights against 6.0 GiB usable, leaving room for 8k tokens. For unquantized weights you need at least a Jetson Orin Nano Super (8GB).
Why is the KV cache for SmolLM2 135M Instruct the size it is?
Because it stores one key and one value vector per token, per layer: 30 layers × 3 KV heads × 64 dimensions × 2 (K and V) × 2 bytes = 22.5 KiB per token. Grouped-query attention shares those 3 KV heads across 9 query heads, cutting the cache by 3× against multi-head attention. None of this can be read off the parameter count.
Compare with
- SmolLM2 360M Instruct258 MiB
- SmolLM2 1.7B Instruct1007 MiB
- Qwen2.5 0.5B Instruct379 MiB
- Qwen3 0.6B433 MiB
- Gemma 3 1B Instruct576 MiB
- MiniCPM5 1B622 MiB
- TinyLlama 1.1B Chat633 MiB
- Llama 3.2 1B Instruct770 MiB
- Qwen2.5 1.5B Instruct940 MiB
- DeepSeek-R1-Distill-Qwen 1.5B1.0 GiB
Method and limits. The parameter count and every architecture figure on this page are read from the model's own published config and the Hub's index over its tensor shapes, fetched 2026-08-07 — nothing here is recalled from a model's name. Weight bytes are the size of a real published file wherever one exists and the parameter count times the published llama.cpp bits-per-weight where it does not; every row says which. Speed figures are roofline bounds — memory bandwidth divided by bytes read per token — not benchmarks. Nothing on this page is written by a language model.