How to Quantize LLM Models: GGUF Explained (2026 Guide)

How to Quantize LLM Models: GGUF Explained (2026 Guide)

A modern 8-billion-parameter language model ships at roughly 16 GB in full precision — too heavy for most laptops and consumer GPUs. Quantization is the compression technique that squeezes those same models down to 4–8 GB with barely any quality loss, and GGUF is the file format that makes them run on your CPU, GPU, or Mac. This guide explains how quantization works, decodes cryptic names like Q4_K_M, and walks you through quantizing a model yourself with llama.cpp.

What quantization actually does

A trained LLM is mostly a giant pile of weights — decimal numbers stored as 16-bit floating-point values (FP16). Quantization converts each weight from a high-precision decimal into a much smaller integer: 8-bit, 5-bit, 4-bit, or even 2-bit. An 8B model that needs 16 GB at FP16 drops to about 8 GB at 8-bit and roughly 4.5 GB at 4-bit.

The trick is that raw rounding would destroy accuracy, so modern quantizers work in small blocks of weights (often 32 weights per block) and store one scale factor per block. Each weight is divided by its block's scale and rounded to the nearest integer. When the model runs, the weights are dequantized on the fly (weight ≈ integer × scale) before the matrix math happens. Per-block scaling preserves accuracy because weights in different layers have very different value ranges.

The quality penalty is surprisingly small. Benchmarks on Llama 3.1 8B show Q4_K_M scoring 69.15 vs 69.41 for Q8_0 on task benchmarks — a gap most users will never notice in chat. Below 4 bits, though, degradation accelerates noticeably: 2-bit quants are roughly half the size of 4-bit ones but the model starts making obvious mistakes.

GGUF: the container format for local AI

GGUF (GPT-Generated Unified Format, maintained by the llama.cpp project) replaced the older GGML format as the standard container for models that run outside datacenters. One .gguf file holds the weights, the tokenizer, and metadata describing the architecture and quantization scheme.

Its main rivals target different hardware:

• GGUF — CPU, hybrid CPU+GPU, and Apple Silicon inference. Used by llama.cpp, Ollama, LM Studio, and Jan. The one format you need for local chat on a personal machine.
• GPTQ / AWQ — Nvidia GPU inference, aimed at datacenter-style serving with tools like vLLM. Pick these on a 24 GB+ consumer GPU when maximum throughput matters.

For running models locally on a Mac, a laptop CPU, or a modest GPU, GGUF is the standard — every guide below assumes it.

Reading a quant name: Q4_K_M decoded

Hugging Face model pages list files like qwen3.5-8b-Q4_K_M.gguf. Each piece of the suffix means something:

• Q4 — bits per weight (4-bit in this case). Lower bits = smaller file = more quality loss.
• K — K-quant method. K-quants use block-wise quantization with mixed precision across different parts of each block, allocating bits more intelligently than the old legacy quants (Q4_0, Q4_1, Q8_0). K-quants are recommended over legacy quants for nearly all hardware.
• M — Medium variant. The suffixes _S (Small), _M (Medium), _L (Large), and _XS (Extra Small) adjust how many sensitive layers are kept at higher precision. _M is the balanced default; _S is smaller and slightly worse; _L is larger and slightly better.
• IQ variants (IQ3_S, IQ2_XXS) — I-quants, a newer family inspired by QuIP techniques. They shrink files further with lookup-table dequantization but cost more CPU during inference.

So Q4_K_M = 4-bit weights, K-quant block method, medium precision variant. It is the community's default recommendation for good reason.

How much smaller do models get?

Real numbers from published benchmarks give a sense of the savings. For a Llama-3.1-style 8B model:

For a 65B model, the same story at a bigger scale: Q8_0 is ~69 GB, Q5_K_M is ~46 GB, and Q4_K_M is ~39 GB. A 70B model that needs about 140 GB in FP16 comes down to roughly 40 GB at 4-bit — the difference between "needs a server" and "fits on one beefy workstation".

Which quant should you pick?

There is one decision rule that covers most people: prefer a bigger model at Q4 over a smaller model at Q8 if both fit your RAM. A 14B model at Q4_K_M will typically outperform a 7B model at Q8_0, because the extra parameters carry more knowledge than the extra precision preserves.

A quick decision tree:

• 8 GB VRAM or 16 GB RAM laptop — 8B model at Q4_K_M. This is the sweet spot for most people.
• Plenty of VRAM, accuracy-critical task (code review, math) — Q8_0, or Q5_K_M as a compromise. Q8_0 is nearly indistinguishable from full precision.
• 32 GB+ RAM, CPU-only — Q4_K_M for the best balance of intelligence and speed; Q5_K_M if you care more about accuracy than generation speed.
• Desperately short on RAM — Q3_K_S or Q2_K, but expect visibly degraded reasoning.

Note that loading a model is an all-at-once affair: you need enough RAM or VRAM for the whole file, plus a few GB extra for the context window (KV cache). Running a 4.6 GB model with a large context still needs noticeably more than 4.6 GB.

Quantize a model yourself: step-by-step

Most people never need to quantize anything — the community publishes GGUF files at every common level on Hugging Face. But if you fine-tuned your own model, here is the standard workflow with llama.cpp.

1. Convert Hugging Face weights to GGUF. llama.cpp ships convert_hf_to_gguf.py. Point it at your model folder and produce an F16 GGUF first:

python3 convert_hf_to_gguf.py /path/to/your-model \
    --outfile model-f16.gguf \
    --outtype f16

(GGUF does not support BF16, so FP16 is the conversion target.)

2. Quantize the F16 file. The llama-quantize binary takes the input file, the output file, and the quant type:

llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

The quantizer analyzes weight distributions, groups weights by sensitivity, and allocates bit budgets per group — sensitive layers keep higher precision, less sensitive ones get fewer bits. Quantizing down from 16-bit directly gives the best result; re-quantizing an already-quantized file (--allow-requantize) compounds the quality loss, so avoid it.

3. Run it. Test the result with the llama.cpp CLI:

llama-cli -m model-Q4_K_M.gguf -n 128

If the output reads coherently, the quant is good. Disk space note: keep the F16 intermediate around until you have verified the quantized file — the F16 is your source of truth for any future re-quant.

Skip the work: download pre-made quants

For standard models, grab a ready-made GGUF from Hugging Face instead of converting and quantizing yourself. Quant uploaders like Bartowski publish nearly every popular open-weight model at Q4_K_M, Q5_K_M, Q8_0, and more. In Ollama you do not even pick a file — just pull the tagged variant:

ollama pull llama3.1:8b-instruct-q4_K_M
ollama run llama3.1:8b-instruct-q4_K_M "Explain quantization in one paragraph."

LM Studio provides the same one-click flow with a desktop GUI, and all of these tools run the same llama.cpp engine underneath — a model validated in one behaves identically in the others.

Advanced quality: the imatrix trick

For the best possible quality at low bit counts, llama.cpp can generate an imatrix (importance matrix) from a representative sample of text. The imatrix tells the quantizer which weights matter most for real-world outputs, so bits are spent where they count.

I-quants (IQ3_S, IQ2_XXS, and friends) take this furthest: with an imatrix, a 2–3 bit model can stay surprisingly coherent. The catch is dequantization cost — I-quants lean on lookup tables during inference, which is heavier on the CPU and varies in speed. Stick with K-quants unless you are deliberately chasing the smallest possible file.

Common mistakes to avoid

• Quantizing from a quant. Always quantize from the F16/32-bit original. Each generation of compression multiplies the error.
• Choosing legacy Q4_0 over Q4_K_M. Legacy quants are only kept for old hardware. For anything modern, K-quants are both smaller and more accurate.
• Ignoring context memory. A long context window (128K tokens) needs gigabytes of KV cache on top of the model weights — the most common reason a "fits in RAM" model still crashes.
• Going below 4 bits for agentic work. Coding assistants and tool-using agents degrade fastest at low precision. Keep them at Q4_K_M or higher.
• Mixing formats. A GPTQ file will not load in llama.cpp and a GGUF file will not load in a GPTQ server. Match the format to your inference stack.

Key takeaways