Does quantization reduce model quality?

8-bit quantization is usually close to lossless; 4-bit methods such as AWQ or GPTQ can cause small drops on some tasks. Always confirm on your own evaluation set.

What is continuous batching?

A serving technique that adds new requests to the GPU batch as others finish, instead of waiting for a whole batch to complete, which greatly increases throughput.

Which LLM serving engine should we use?

vLLM is a strong default for open-weight models; SGLang and TensorRT-LLM can be faster for specific setups. Benchmark with your own traffic.

Can we keep an OpenAI-compatible API when self-hosting?

Yes. Most modern serving engines expose OpenAI-compatible endpoints, so applications can switch models with minimal code changes.

LLM Inference Optimization: How to Cut Cost per Token Without Losing Quality

LLM inference optimization: quantization, continuous batching, prefix caching, speculative decoding and GPU right-sizing to cut cost per token.

LLM inference cost is driven by GPU time per token. The biggest savings usually come from a modern serving engine with continuous batching, quantization to 8-bit or 4-bit, prefix caching for repeated prompts, the smallest model that meets your quality bar, and autoscaling GPUs to real traffic. Measure cost per million tokens and latency percentiles before and after every change.

What drives LLM inference cost?

Every generated token requires a pass through the model, and GPUs are expensive whether they are busy or idle. Cost therefore depends on model size, how many requests share each GPU at once, how long prompts and outputs are, and how much idle capacity you pay for.

Latency has two parts users notice: time to first token, dominated by processing the prompt, and time between tokens, dominated by generation speed and contention.

Which optimisation levers matter most?

In rough order of typical impact for self-hosted models:

How do you choose the right GPU?

Choose by memory first: the model weights, the KV cache for your concurrency and context length must fit. Larger GPUs such as H100 or H200 suit big models and high throughput; L40S and L4 class cards are cost-effective for smaller or quantized models.

Mix capacity types: reserved capacity for steady base load, on-demand or spot capacity for peaks and batch jobs. Scale-to-zero is useful for internal tools with idle periods.

Should you self-host or use an API?

Managed APIs are simplest and often cheapest at low or spiky volume. Self-hosting open-weight models becomes attractive when volume is high and steady, when data must stay inside your perimeter, or when a fine-tuned small model can do the job.

Many teams use both behind an AI gateway that routes each request to the cheapest model that meets its quality requirement, with fallbacks when a provider is slow or unavailable.

How do you measure improvements?

Benchmark with realistic traffic — your actual prompt and output lengths and concurrency — not synthetic single requests. Track throughput (tokens per second per GPU), time to first token, P95 latency, GPU utilisation and cost per million tokens.

Run your quality evaluation after every optimisation. Quantization and smaller models are only savings if answers stay correct.

A pragmatic optimisation plan

Start with measurement, then change one lever at a time:

Right-size the model: a well-prompted or fine-tuned smaller model often matches a larger one on a specific task.
Use a serving engine with continuous batching and paged attention, such as vLLM, SGLang or TensorRT-LLM, so many requests share the GPU efficiently.
Quantize weights (AWQ, GPTQ or FP8) to reduce memory and increase throughput with little quality loss — verify on your own evaluation set.
Enable prefix and KV-cache reuse when many requests share the same system prompt or documents.
Try speculative decoding, where a small draft model proposes tokens that the large model verifies.
Trim prompts: shorter system prompts and tighter retrieved context reduce work on every request.
Profile current traffic and set cost and latency targets.
Move to a modern serving engine and enable batching and caching.
Test quantized and smaller models against your evaluation set.
Right-size GPUs and configure autoscaling.
Add a gateway for routing, caching and cost dashboards.
Our LLM inference and GPU infrastructure services cover this work end to end.