Four-Bit Weights Do Not Guarantee Cheaper Inference

A model fits on the accelerator after its weights are reduced to four bits. That is a useful result, but it answers only one question. It does not establish that requests finish sooner, that more useful work completes per pound, or that the model still handles the difficult cases that made the original deployment worthwhile.

The release decision is whether a specific numerical recipe, running on a specific serving stack, buys enough capacity or speed without crossing the application's quality limits. The smallest checkpoint is not necessarily the best service.

This guide concerns teams serving their own LLM checkpoints. Primary documentation and research explain the mechanisms below; the release gates and worked arithmetic are AIEngine's engineering analysis, not results from an AIEngine benchmark. Keep three claims separate throughout the review: smaller storage, better serving performance and acceptable behaviour.

Write a numerical [contract](/industries/legal), not a bit-width label

Quantisation represents values using a restricted set of numerical levels. NVIDIA's accuracy guidance distinguishes rounding to a representable value from clamping values outside the chosen range. Widening that range can reduce clipping while making neighbouring levels farther apart. Integer and floating-point formats distribute their representable values differently; INT4 and FP4 are not interchangeable recipes merely because both use four bits.

Describe which tensors change. W4A16 commonly denotes four-bit weights with sixteen-bit activations, not a model in which every operation, accumulator and intermediate uses four bits. Check the implementation rather than extending that shorthand into an arithmetic guarantee. TensorRT's supported quantised types, for example, explicitly identify INT4 as weight-only quantisation.

Give the reviewer a compact numerical contract:

• Weight and activation formats, plus accumulation precision where configurable.
• Scale granularity, group size and whether zero points are stored.
• Layers retained at higher precision and the reason for each exception.
• KV-cache format and any attention-backend requirements.
• Quantisation method, converter version and exact source checkpoint.

This record prevents two candidates labelled four-bit from being treated as equivalent. It also makes a later regression diagnosable: a changed group size, skipped layer or cache setting is a release change, even when the marketing label stays the same.

Account for bytes that do not disappear

Start with an intentionally simplified calculation. Eight billion dense weight values stored at sixteen bits occupy 16 GB of raw payload. At four bits, that payload becomes 4 GB. These are decimal gigabytes, not GiB, and neither figure is a measurement of a running process.

Suppose the four-bit representation additionally stores one sixteen-bit scale for every sixty-four weights. There are 125 million groups, so scales add 0.25 GB. The resulting 4.25 GB is already larger than the headline payload, before zero points, padding, higher-precision tensors or runtime allocations. The weight-store reduction in this illustration is about 3.76 times, not a fourfold reduction in total application memory.

Measure peak device memory under the intended context lengths and concurrency. Include model loading, compilation or graph capture, temporary workspaces and steady-state serving. A checkpoint that loads successfully can still exhaust memory when requests arrive together.

The HBM and inference-memory guide explains why weights and request state belong in different capacity calculations. Quantising weights does not automatically quantise the KV cache. vLLM documents KV-cache quantisation as a separate configuration, including scale-calibration choices and backend-specific behaviour. Treat that as a separate experimental change; otherwise a combined gain or regression cannot be attributed cleanly.

Keep a memory reserve rather than filling every recovered byte with additional requests. The operational benefit may be fewer out-of-memory failures or a longer supported context, even if per-request latency does not improve.

Choose a recipe your runtime can actually execute

Low-bit storage is useful only if the serving implementation can exploit it. Packing, scale handling, conversion and kernel selection all participate in the result. The vLLM compatibility documentation makes support conditional on the quantisation implementation and hardware. Check the matrix and code for the version being deployed, not just the continuously updated documentation page.

Two original methods illustrate why recipe names matter. AWQ, first submitted in June 2023, uses activation statistics to identify salient weight channels and applies scaling to reduce low-bit weight-quantisation error. Its argument is not that every weight contributes equally to the outcome.

SmoothQuant, first submitted in November 2022, instead describes an equivalent rescaling that shifts quantisation difficulty from activation outliers towards weights, enabling its W8A8 approach. These are different interventions, not successive rungs on a single quality ladder. Their published results do not establish a winner for an arbitrary fine-tune or serving workload.

Shortlist only recipes supported by the intended model architecture, hardware and maintained runtime. Confirm the actual loaded precision and selected execution path from runtime diagnostics. A successful load proves compatibility at that boundary; it does not prove that the intended fast kernel ran.

Reject a candidate whose only advantage requires an unsupported custom kernel that the team cannot maintain. Conversely, do not discard a higher-bit candidate before measuring it: its available execution path may better match the workload.

Treat calibration as part of model construction

For a recipe that estimates scales or activation statistics from examples, calibration data helps construct the candidate. It is not independent evidence that the resulting model is ready to release.

NVIDIA's quantisation-workflow documentation distinguishes post-training quantisation from quantisation-aware training and describes tensor, channel and block granularity. Finer-grained scales can improve representation at the cost of additional metadata and processing. Quantisation-aware training introduces simulated quantisation during training; it is a different engineering commitment, not a flag that guarantees preserved quality.

Build calibration examples from the shapes of inputs the service actually handles. A conversational sample can be a poor stand-in for long retrieved documents, source code, multilingual text or strict tool schemas. Preserve the tokenizer, chat template, truncation policy and sampling procedure with the dataset revision. Otherwise “the same calibration set” can produce different effective inputs.

Keep three collections distinct:

• Calibration material used to derive the numerical representation.
• Development cases used to choose recipes and diagnose failures.
• A held-out release set used after the choice is frozen.

Repeatedly selecting a recipe on the release set turns it into another development set. Retain a fresh evaluation reserve and document how often selection used each collection. If privacy restrictions prevent representative calibration, record that limitation and test the affected input slices explicitly; do not silently substitute a convenient public dataset and call it representative.

Benchmark prompt processing and generation separately

An inference service processes prompts before generating tokens. A workload dominated by long inputs can behave differently from one dominated by short prompts and long outputs. Benchmark both parts, then the complete request path.

A dated example shows the danger of borrowing a speedup. The PyTorch team's January 2025 GemLite, TorchAO and SGLang benchmark reports INT4 weight-only decode speedups of 1.95 times at batch size one and 1.16 times at batch size thirty-two, both with tensor-parallel size one, for its Llama 3.1-8B setup on H100 hardware. Its baseline was a compiled bfloat16 model, and accuracy was explicitly outside the article's scope. Those are experimental conditions, not forecasts for another deployment or recommendations to install that historical software stack.

Run the baseline and candidate with the same model revision, prompts, decoding policy, request arrival pattern and output limits. Record any optimisation that cannot be held constant. Separate cold-start costs from warmed operation, and measure queueing rather than timing only the model call.

At minimum, retain:

• Time to first token, inter-token latency and complete-request latency.
• Tail latency at the intended concurrency, not only averages.
• Peak device memory and the largest stable workload envelope.
• Completed requests, failures, cancellations and generated token counts.
• Accelerator time and other attributable costs per accepted task.

Do not let early termination look like an efficiency win. A candidate that emits shorter but incomplete answers can produce attractive latency figures. Compare useful completions against the same task contract, and include retries, fallback inference and additional review in the cost calculation.

Make quality loss visible at task boundaries

Start with a reference model that already meets the application's requirements. Agreement with that reference is a diagnostic, not ground truth: two variants can make the same mistake. Evaluate both against independently labelled answers, executable tests or reviewed task criteria.

For document extraction, distinguish valid JSON from correct fields. A syntactically perfect amount with the wrong currency remains wrong. For a coding assistant, compilation is weaker than passing the relevant tests. For a tool-using model, a valid function name does not establish that its arguments or authority are appropriate. The output-validation guide supplies the downstream action boundary; numerical optimisation must not weaken it.

Set both an absolute quality floor and an allowed regression relative to the reference. Break results into consequential slices: long contexts, uncommon languages, decimal quantities, negation, missing information and refusal-required inputs, as relevant to the product. Quantisation does not necessarily damage all these slices; the point is to make a concentrated regression observable instead of assuming an aggregate score will reveal it.

Use paired cases so reviewers can inspect where outcomes changed. Report sample counts and uncertainty alongside pass rates. Zero observed failures on a tiny set is weak evidence about a rare error. Where an error is unacceptable, keep deterministic validation, escalation or human review in the production path rather than claiming that testing proves it impossible.

Perplexity and generic benchmarks can detect broad problems quickly. They should not replace the decision-specific tests that determine whether this service remains useful.

Work the release decision without inventing a win

Consider an illustrative document-extraction service that proposes fields for review but does not execute payments or update records autonomously. Its owner wants to support more simultaneous long documents on the existing accelerator. Compare the approved sixteen-bit reference with one supported weight-only candidate, keeping cache precision unchanged initially. This is a test design, not a reported deployment.

Before running it, the owner defines acceptable review accuracy, latency limits, memory reserve and minimum capacity improvement. The quality owner identifies fields that require exact matching and errors that always require escalation. Do not choose those thresholds after seeing which candidate looks best.

The decision table deliberately contains no invented benchmark numbers. A release requires measured values attached to the declared thresholds. “Fits now” can be a legitimate objective when the previous configuration could not serve the workload, but then compare against a viable alternative deployment rather than presenting an unusable baseline as a cost saving.

Ship the recipe and keep the comparison alive

Promote the evaluated bundle, not a similarly named download. Preserve the base and converted artefact digests, numerical contract, calibration revision, runtime and driver versions, hardware identity, benchmark inputs and acceptance record. The model-artifact admission guide covers the security checks around loading and conversion; passing numerical evaluation does not replace those checks.

Use a limited canary with the reference configuration available for rollback. Watch the same quality slices and complete-task costs that justified approval, alongside memory pressure and latency tails. A global success rate is insufficient if one document family becomes harder to review.

Revisit the comparison when the model, tokenizer, prompt template, converter, kernel, driver, cache configuration or workload distribution changes. Some changes may improve the candidate; others can invalidate the original advantage. Record which evidence needs rerunning rather than carrying an old speedup into every subsequent release.

The durable result is not a four-bit badge. It is an evidence-backed statement that this exact representation improves a defined service, within a tested operating envelope, while preserving the behaviours its users depend on.