Playground · app-grounded instrument
On-device AILLM VRAM & KV-Cache Footprint Calculator
Exact memory math for open LLMs (DeepSeek-R1, Llama 3.3, Qwen 2.5, Mistral). Calculate weight sizes, KV-cache scaling up to 128K context, activation overhead, and hardware fit across Apple Silicon, RTX GPUs, and Jetson dev kits.
Already solved for 62 models and 22 accelerators. If your model and card are common ones, the arithmetic below has been done from the model's own config — browse the fit matrix rather than typing head counts in by hand.
Model & Architecture Configuration
Deterministic MathMemory Footprint Breakdown
45.0 GB VRAMHardware Target Compatibility Matrix
Anatomy of LLM Memory Allocation
Running an open LLM on consumer GPUs or Apple Silicon requires partitioning memory across three distinct structures:
Quantized Weight Tensor
The model parameters stored on disk. Quantization packs FP16 weights down to 4-bit (0.56 B/param) or 8-bit (1.06 B/param). Stays resident in VRAM/Unified RAM continuously.
Key-Value Attention Cache
Stores intermediate key and value states per layer for every prompt token to avoid re-computing attention during generation. Grows linearly with context length and batch size.
CUDA/Metal Runtime Context
GPU driver context buffers, scratch spaces for FlashAttention, and activation tensors. Typically consumes 1.0 GB to 2.5 GB depending on framework (vLLM, llama.cpp, CoreML).
Gear & Hardware Picks for Local LLMs
Tested studio gear for running large models on-device and high-throughput inference.
$61.00BookDeep Learning (Adaptive Computation and Machine Learning series)
Foundational deep-learning textbook referenced while building itria.
$40.00BookDesigning Machine Learning Systems: An Iterative Process for Production-Ready Applications
ML systems-design reference used while building itria.
$49.50BookHands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems
Practical ML reference used while building itria.
$129.99AcceleratorGoogle Coral USB Edge TPU ML Accelerator coprocessor for Raspberry Pi and Other Embedded Single Board Computers
USB Edge TPU for int8 quantized nets — run the same quantized CoreML model sized here and see why int8 cuts RAM bandwidth but needs per-channel scales.
$72.22BookProgramming Massively Parallel Processors: A Hands-on Approach
CUDA/GPU parallel programming text for WebGPU PINN and edge GPU workloads.
$259.99SBCJetson Nano Developer Kit 16G eMMC onboard for AI Machine Learning (4GB RAM 16GB eMMC)
eMMC variant for TinyML deployment — flash int8 quantized CoreML model sized by this calculator and measure flash vs RAM footprint tradeoff.
$383.99SBCNVIDIA Jetson Nano Developer Kit (945-13450-0000-100)
Edge GPU where int8 quantized models from this calculator actually run — compare theoretical size saving vs measured latency drop on Jetson vs iPhone Neural Engine.
$434.97SBCNVIDIA Jetson Orin Nano Super Developer Kit
67 TOPS edge AI dev kit — benchmark int4 quantized models sized here and validate that CoreML quantized size math predicts actual flash/RAM usage on device.
$269.99StorageSANDISK 1TB Extreme Portable SSD (New Model) - up to 2000MB/s Transfer speeds, USB Type-C connectivity, Reliable Durability - Black - SDSSDE70-1T00-G25
Portable SSD used for studio project storage and backups.
$259.95SBCCanaKit Raspberry Pi 5 Starter Kit PRO — Turbine Black, 8GB RAM, 128GB
Flagship Pi 5 8GB board — Amazon verified ASIN B0CK2FCG1K (via DuckDuckGo Amazon search). SparkFun third-party gave no commission; now Amazon affiliate.
$399.99SBCSeeed Studio Raspberry Pi 5 Starter Kit - 16GB RAM, 64GB SD, Type-C PSU, Active Cooling Case
Pi 5 16GB starter kit — Amazon verified ASIN B0F944X9S4 (Seeed Studio 16GB + 64GB SD, Type-C PSU, case). 14.5 GiB ceiling for Q4_K_M up to ~12B params. SparkFun third-party gave no commission; now Amazon.
$1109.99SBCYahboom Jetson Orin NX Super 16GB RAM 157 TOPS Dev Kit JetPack 6.2 256GB SSD
Orin NX 16GB — 100 TOPS unified LPDDR5, 14.5 GiB model ceiling, JetPack 6.2 + 256GB SSD included. Amazon verified — runs TensorRT-LLM for 4-12B Q4_K_M at 4k+ context.
$353.99SBCOrange Pi 5 Plus 16GB RK3588 8-Core SBC, 2.4GHz, 8K Video Decoding
RK3588 8-core 16GB LPDDR4X, 14 GiB model ceiling — verified Amazon ASIN B0GYCTT6YM, affordable Pi5-class host for llama.cpp / RKLLM with NVMe.
Radxa ROCK 5B - 16GB
RK3588 ROCK 5B 16GB LPDDR4X — 14 GiB ceiling, PCIe NVMe, strong llama.cpp RKNN target. Official Radxa product page verified (no Amazon affiliate SKU yet).
LattePanda Sigma - 32GB (DFRobot DFR1080)
Intel Core i5-1340P 12C/16T + 32GB LPDDR5, 30 GiB model ceiling — verified DFRobot SKU DFR1080 product-2671.html, fits 15.8 GiB GGUF (Gemma 26B Q4_K_M ≈19.7 GiB runtime) with headroom. Uses ?tracking_id=vwfcds.
$29.69BookTinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers
Quantization-aware training fp32→int8 and model footprint math — same byte-size arithmetic bytes = params * bits/8 this calculator does for CoreML fp16/int4.
$6,999.00ComputerApple MacBook Pro M5 Max, 128GB Unified Memory, 2TB SSD
The portable 128 GB machine. Same resident-model argument as the Mac Studio, in a laptop you can profile on.
$1,699.00ComputerApple MacBook Air 13-inch M5, 32GB Unified Memory, 512GB SSD, Midnight
32 GB of unified memory in the lightest Apple silicon body — enough to keep a quantized mid-size model resident instead of streaming it off SSD.
$329.00WearableApple Watch Series 11, GPS 46mm, Jet Black Aluminum
The watchOS target itself. Any on-device inference claim for the Watch is bounded by its CPU-accessible bandwidth, which Apple does not publish.
$4,649.99ComputerNVIDIA DGX Spark — Personal AI Desktop Supercomputer, GB10 Grace Blackwell
128 GB of coherent unified memory on a GB10 Grace Blackwell chip. A 70B at Q4_K_M is ~33 GiB of weights, so this holds one resident with room for long context — and Q8 too.
$4,549.99ComputerASUS Ascent GX10 Personal AI Supercomputer — GB10, 128GB LPDDR5x, 2TB NVMe
The same GB10 superchip and 128 GB unified memory as the DGX Spark, on a 2 TB NVMe. Sold explicitly as a local-LLM and RAG workstation.
$12,855.95GPUPNY NVIDIA RTX PRO 6000 Blackwell Workstation Edition — 96GB GDDR7
96 GB of GDDR7 on one card. The single-GPU route to a resident 70B: the weights fit roughly three times over at Q4_K_M, and the memory bandwidth is what actually sets decode speed.
$12,950.00GPUPNY NVIDIA RTX PRO 6000 Blackwell MAX-Q Workstation Edition — Dual Fan, 96GB GDDR7
The Max-Q variant of the 96 GB card — same memory, a lower power envelope, for a workstation that cannot feed a 600 W board.
Prices shown were retrieved from the Amazon Product Advertising API on 19 July 2026 and are indicative only — the price and availability on Amazon at the time of purchase apply.
Prices shown were checked against the Amazon product listing on 9 August 2026 and are indicative only — the price and availability on Amazon at the time of purchase apply.
Build this lab
Local vs cloud inference bench
Jetson Orin Nano for local GPU inference + Pi 5 as an offload gateway — reproduce the client-vs-serverless tradeoff on real hardware before you commit to a cloud GPU bill.
- $435
- $260
- $130
- $270
- $40
Prices shown were retrieved from the Amazon Product Advertising API on 19 July 2026 and are indicative only — the price and availability on Amazon at the time of purchase apply.
Prices shown were checked against the Amazon product listing on 9 August 2026 and are indicative only — the price and availability on Amazon at the time of purchase apply.
Estimated total
$695
Prices from Amazon catalog cache · may change
The Mathematics of Memory & Bandwidth
The total VRAM footprint M_total is governed by:
Autoregressive token generation speed T_gen (tokens/sec) is memory-bandwidth limited during decoding:
Inference Startup & Allocation Command
Copyable startup template for vLLM / llama.cpp with exact memory ceilings.
# Python / vLLM / llama.cpp startup script generator
# Run with estimated memory ceiling
python3 -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen2.5-72B-Instruct-GGUF \
--gpu-memory-utilization 0.90 \
--max-model-len 32768 \
--kv-cache-dtype fp8Export · Soft gate
Export vLLM & llama.cpp Benchmark Config Script
Free watermarked file instantly. Unlock clean export via email — soft gate, no hard paywall. We store unlock in localStorage per sim.
File · llm-vram-kvcache-calculator.txt
llm-vram-kvcache-calculator.txttext/plain+ watermark line on free pathFree download adds a small footer:
# Export from makerportal.ai — free watermarked build. Unlock c…Clean export removes footer. Both are generated fresh from your current sim tuning.
Privacy: email stays in your browser localStorage (mp_export_email_llm-vram-kvcache-calculator) + unlock flag (mp_export_unlock_llm-vram-kvcache-calculator). If Buttondown username is configured, we also POST to Buttondown (privacy-first mode, no tracking pixels per D-014). See privacy → affiliates & email.
Unlock clean export
Soft gate — no hard paywall, no Clerk. Email stays local unless you explicitly check the newsletter box. Unsubscribe anytime. RSS at /rss.xml.
✓ Lab Pro — clean export on every lab
Your licence unlocks this and every other gated simulator, so there is nothing to enter here. Manage or sign out on the shop page.
✓ Unlocked — clean exports enabled
Stored in mp_export_unlock_llm-vram-kvcache-calculator. Clean file omits watermark. Re-lock via browser devtools → localStorage.
Frequently Asked Questions
How is total LLM VRAM requirement calculated?↓
Total VRAM is the sum of model weight footprint, KV-cache residency, activation memory, and runtime overhead: VRAM = (Params * BytesPerWeight) + (2 * Layers * Heads * HeadDim * SeqLen * Batch * BytesPerKV) + ActivationOverhead + RuntimeContext. For unified memory (Apple Silicon), this entire memory pool resides in system RAM shared with macOS.
What is the KV-Cache memory impact at 128K context lengths?↓
At 128,000 sequence length, the KV cache often exceeds the model weights themselves! For a 70B model in FP16 KV-cache, a single request at 128K tokens requires over 32 GB of KV-cache alone. Utilizing FP8 or INT4 KV-cache quantization reduces this footprint by 2x to 4x with minimal accuracy degradation.
Why does Apple Silicon (M1–M4 Ultra) perform so well on large models?↓
Apple Silicon uses Unified Memory Architecture (UMA) with extreme bandwidth (up to 800 GB/s on Ultra chips). Instead of being capped at 24 GB on consumer PCIe GPUs, a Mac Studio with 192 GB Unified Memory can run a 70B FP16 or DeepSeek-R1 Q4 model natively within one shared memory pool.
What token generation speed (tokens/sec) can I expect?↓
Memory bandwidth is the primary bottleneck during LLM autoregressive decoding. Generation speed is approximately Memory Bandwidth (GB/s) divided by Total Active Model Weight Size (GB). On a 400 GB/s M3 Max running a 20 GB Q4 model, maximum theoretical speed is ~20 tokens/sec per stream.