How quantization affects your Cargo

Quality tiers, model families, and what quantization means for the Cargo you run on Archipelag.io.

How quantization affects your Cargo

Large language models are shipped in different quantizations — compressed formats that trade a little accuracy for dramatically smaller size and faster inference. A 7B model at full precision needs ~14 GB of memory; the same model at Q4_K_M fits in under 5 GB and runs on a laptop or a phone.

Archipelag.io treats quantization as a first-class property of every LLM Cargo, so you always know what you’re getting.

Model families and variants

A model family (e.g. Llama 3.1 8B Instruct) is one underlying model. Each family owns several variants — the same model at different quantizations (Q5_K_M, Q4_K_M, IQ3_M, and so on). Variants share the family’s identity and license but differ in size, speed, memory footprint, and quality.

When you run a family, placement automatically picks the best variant for the Island that will serve you: a roomy machine lands a heavier, higher-quality quant, while a memory-constrained phone falls back to a lighter one — same model, appropriate variant.

Quality tiers

Every variant carries a quality tier so you can judge accuracy at a glance:

The tier badge appears on each Cargo card in the Cargo Registry. Standard-tier variants are the sweet spot for most work; compact and experimental variants are cheaper and lighter but can degrade on demanding tasks (coding, math, long-context reasoning).

Confidential jobs enforce a quality floor

Confidential Cargos — those that run only on hardened, attested Islands — enforce a minimum quality tier of standard. A confidential job will not silently run on an aggressively-degraded quant, and a request that would do so is rejected with a clear error rather than returning low-quality output you didn’t ask for.

You can always opt up (request reference for medical or legal work); you cannot opt below the floor for a confidential job. Commodity (non-confidential) jobs may use any tier.

Choosing a variant

• General chat, drafting, summarization — standard (Q4_K_M / Q5_K_M) is the default sweet spot.
• Cost- or memory-sensitive, on-device — aggressive or experimental quants unlock bigger models on smaller hardware; expect some quality loss.
• High-stakes / precision-critical — reference where an Island has the memory to serve it.

The canonical accuracy numbers (MMLU, HellaSwag, and others) for each variant are published on the family’s detail page as they’re evaluated.